freebsd.munk.me.uk - Serendipity

Customizing FreeBSD Port Installations With Patch Files

nospam@example.com (munk) — Thu, 07 Sep 2006 01:29:28 +0000

-
This article presumes the reader has a good working knowledge of the FreeBSD ports system. For more information see the ports manpage, the FreeBSD handbooks section on ports and the FreeBSD porters handbook.
-

The FreeBSD ports tree is great, but when it comes to making changes to the files that are installed by a port you have to be careful so as not to lose any changes when you upgrade. Say you install a port like the Serendipity weblog software, then go to modify a template. Everything works great but unfortunately when you next use portupgrade you find the changes you made were all clobbered by the upgrade! Not good.

There are a few solutions to this problem, but probably the most natural method - in terms of keeping with the ports system and your own changes in harmony - is to create your own custom patch files and place them inside the port's 'files' directory. This way when you install or upgrade a port, the files you want patching to include your modifications will automatically be patched.

Here's a simple example using this weblog, Serendipity. I want to modify the template I'm using so it includes a Google Analytics tracker in each and every page to help track web usage. The easiest way to do this is to modify the index.tpl file so it includes the analytics tracker code just before the closing </body> tag of the index.tpl file.

To make the changes permanent after an upgrade, instead of just going ahead and modifying the 'live' index.tpl that's already installed I'm going to make the changes to the index.tpl file located in the serendipity ports working directory 'work' and then create a patch file that will transform the original index.tpl to the modified index.tpl.

To do this step by step:

Ensure the working directory for serendipity is available - to do this run 'make extract' to extract the distfile's contents into the 'work' directory:

CODE:
root@users /home/munk/ports/www/serendipity# cd /usr/ports/www/serendipity/
root@users /home/munk/ports/www/serendipity# make extract
===> Found saved configuration for serendipity-1.0.1
===> Extracting for serendipity-1.0.1
=> MD5 Checksum OK for serendipity-1.0.1.tar.gz.
=> SHA256 Checksum OK for serendipity-1.0.1.tar.gz.
Move into the template directory and create a copy of the file you want to modify, naming it <file>.orig, where <file> is the name of the file to be modified:

CODE:
root@users /home/munk/ports/www/serendipity# cd work/serendipity/templates/contest/
root@users /home/munk/ports/www/serendipity/work/serendipity/templates/contest# cp index.tpl
index.tpl.orig
Make the changes to the file you're modifying - note NOT the .orig file! - in this case I need to make changes to index.tpl to add the following before the ending tag:

CODE:
<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-XXXXXX-1";
urchinTracker();
</script>
Create a patch file in the 'files' directory of the port containing the unified diffs between the original file and the modified file.
Notes:
1. The name of the patch file MUST begin with 'patch-' to be treated as a patch file. The rest of the filename should reasonably indicate what the patch file contains. In my case I'll call the patch file 'patch-templates_contest_index.tpl'.
2. The patch must be made relative to the directory into which the port's distfile was extracted - if in doubt check over other port's patch files to see how they do it!
3. Make sure the .orig file is first in the diff command
CODE:
root@users /home/munk/ports/www/serendipity/work/serendipity/templates/contest# cd ../..
root@users /home/munk/ports/www/serendipity/work/serendipity# diff -u templates/contest/index.tpl.orig templates/contest/index.tpl > usr/ports/www/serendipity/files/patch-templates_contest_index.tpl
root@users /home/munk/ports/www/serendipity/work/serendipity# cat /usr/ports/www/serendipity/files/patch-templates_contest_index.tpl
--- templates/contest/index.tpl.orig Thu Sep 7 03:13:56 2006
+++ templates/contest/index.tpl Thu Sep 7 03:14:08 2006
@@ -52,6 +52,13 @@
<div id="copyright">Design <a href="http://garv.in/">Garvin Hicking</a>, Icons
<a href="http://tango-project.org/">Tango Project</a></div>

{if $is_embedded != true}
+<script src="http://www.google-analytics.com/urchin.js"
+type="text/javascript">
+</script>
+<script type="text/javascript">
+_uacct = "UA-669467-1";
+urchinTracker();
+</script>
</body>
</html>
{/if}

Now if all went well we should be able to run portupgrade on the installed port - or just run 'make install' in the port directory if it's not installed yet - and the modifications to the index.tpl file should be made automatically.

Admittedly this is a slightly complicated process and there are other easier ways of keeping a file after an upgrade - for example modify pkgtools.conf to make portupgrade execute commands after the port is installed to copy over the files you want updating. However this method isn't as reliable as the patch file method. If any changes are made to the files you're overwriting, portupgrade will never let you know about it - whereas with the patch file method, if one of the files you're wanting to modify has changed in the distribution tarball, then in all likelyhood the upgrade will fail becaue the patch process fails. This allows you to look at the files that have changed in the distribution and change your patch file appropriately.

Google Sitemaps Generator Plugin

nospam@example.com (munk) — Sun, 03 Sep 2006 14:29:25 +0000

Getting back into blogging again after a year or more break has been made a lot more interesting by the huge number of cool plugins that users are submitting to the spartacus s9y plugin site. Spartacus in itself needs a seperate blog entry really being the cool bundle of cool that it is! Anyway for now this entry's about the Google Sitemap Generator Plugin (GSG).

Google Sitemaps are a way for webmasters to get their content indexed by Google more quickly and efficiently. Using the sitemap protocol, webmasters can outline what content is contained on their site and where Google can expect to find that content. This way Google should be able to index a sitemapped website quicker.

There are various ways to create a sitemap ranging from doing it by hand in a text editor or more easily by having the sitemap created automatically for you by a script. Once the sitemap is ready it can be submitted to Google online ready for Google to verify the sitemap and then crawl your site.

All of this is good, except every time the site changes the sitemap becomes outdated. However the Google Sitemap Generator plugin counters that problem. The GSG plugin acts on the publish and save event hooks, so that any time you publish or save an article, the GSG plugin works in the background to create an updated map of all the published content on the site. The plugin even 'pings' Google to let it know that the sitemap has changed, making new content on your site get listed on Google quicker. Ace!

I'd love to give credit to the author but all I know or can find out about him is his name is Boris. Anway, thanks a load Boris :)

EDIT: As I saved this entry the sitemap was generated successfully!:

CODE:

; ls -al /web/freebsd.munk.me.uk/sitemap.xml.gz
-rw-r--r-- 1 www www 40269 Sep 3 15:53 /web/freebsd.munk.me.uk/sitemap.xml.gz

An important note - you do still have to submit the sitemap manually for the first time - a note appears at the top of the s9y editing screen saying: If you have not submited your sitemap to google, do it now with visiting this link. The 'link' seems to be customized to submit the sitemap to google automatically... let's click it... ok yes it goes to the Google webmaster tools site where you have to submit the sitemap manually the first time. All done! Hopefully that'll make it easier for Google to spider the articles.

Serendipity Spam Statistics

nospam@example.com (munk) — Thu, 31 Aug 2006 13:16:31 +0000

I just downloaded this great looking spam statistics plugin for Serendipity from Andreas. Unfortunately after installing it it didn't seem to work, so I got stuck in to see what was up.

Turns out it only works when the spamblock plugin logs to the database, so I'll either look into making it work with log files or maybe think about adding something to the admin stats plugin if that's possible. Or do neither given it's not uber important to me given I get a raft of info on the spam stats each night via a cron job.

I have a cron job that checks various things spam related on a daily basis - checking for referer spam, quarantined files uploaded via PHP, mod_security log entries that need attention and finally checking for serendipity / weblog spam. The situation with weblog spam had gotten so bad on the old domain munk.nu that I even ended up creating a script to convert spamblock log entries into firewall rules for ipf. I'm not kidding, at least 100 trackback spam entries per day through June and July - for the year 2006 so far there are nearly 9000 unique IPs dropping new trackback spam.

What's annoying too is that even adding offending IPs to my firewall block list, each and every new day there would be another 100 new unique IP addresses spamming the blog. No doubt this is a botnet - 100 new zombies found per day sounds like a professional organisation.

Ho hum. Anyway I'll add the 'log2ipf.pl' perl script in the extended part of this article. It's a perl script that's little more than an extended 'grep | sed' which searches for text in a file and then reports how many results it found for each item. In the default case using just 'log2ipf.pl somefile.log' it searches for:

CODE:

"s9y"=>qr/.*\[REJECTED: [No API-created comments|Trackback URL invalid|Filtered by Akismet\.com].*, IP (.*?)].*/,

in this case it reports a list of IP addresses and how many times each IP address was 'caught' trying to spam - but it could be modified to do anything. For example I have another 'filter' setup to see how many people use a google search to find pics on my server by searching for the term 'picasa.ini':

CODE:

"picasa" =>qr/^.*?\s+(.*?)\s+.*%22index\+of%22\+%2F\+picasa\.ini.*/

so I can feed apache logfiles to log2ipf.pl using this commandline:

CODE:

; log2ipf.pl -l picasa /var/log/httpd/all/2006/07/*/*
24.242.97.20: 1
67.141.28.129: 1

telling me there was just 2 such searches during July 2006 (woo). I seem to remember that search returning more than that at the time I wrote the filter though lol. You get the idea anyway.

To add a new 'filter', best thing to do is import a sample logfile line you want to produce a result, then customize the script %re variable to include your custom filter.

For example, say you wanted to search for auth log failures for SSH (this is actually done for you by the periodic utility on FreeBSD if you set it up in /etc/periodic.conf, but that's another article! - you could write something like this for the %re filter:

CODE:

my %re=(
"s9y"=>qr/.*\[REJECTED: [No API-created comments|Trackback URL invalid|Filtered by Akismet\.com].*, IP (.*?)].*/,
#Example of logfile line we want to catch:
# Aug 26 14:57:35 users sshd[30136]: Failed password for root from 211.48.62.102 port 50706 ssh2
"ssh" =>qr/.*Failed password for .* from (.*?) .*/,
"picasa" =>qr/^.*?\s+(.*?)\s+.*%22index\+of%22\+%2F\+picasa\.ini.*/
);

which would result in:

CODE:

; log2ipf.pl -l ssh /var/log/auth.log
168.126.71.148:         1
   210.34.14.53:         3
  84.10.149.105:         3
  211.48.62.102:         3
220.231.54.232:         3
   195.10.193.4:         5
213.179.181.26:        11

As I say you can do the equivalent with grep, sed, sort and uniq on the commandline:

CODE:

; grep "Failed password for" /var/log/auth.log | sed -e 's/.*Failed password for .* from $[^ ]*$.*/\1/' \
  | sort | uniq -c | sort -n
   1 168.126.71.148
   3 210.34.14.53
   3 211.48.62.102
   3 220.231.54.232
   3 84.10.149.105
   5 195.10.193.4
  11 213.179.181.26

But for a very large file the timing differences between this method and the perl script are massive.

Anyhoo this is turning into a crazy long entry so I'll turn it in. The script log2ipf.pl - should rename that really since it's got little to do with ipf really! - is in the extended article below if anyone's interested.

Continue reading "Serendipity Spam Statistics"