freebsd.munk.me.uk

Command Line Web Browsing With WWW::Mechanize::Shell

nospam@example.com (munk) — Wed, 15 Aug 2007 12:46:00 +0000

Introduction
The perl module WWW::Mechanize::Shell is a brilliant tool for browsing websites at a very low level - think somewhere in between using telnet and using a command line based browser like lynx or links or w3m and you'll be close. WWW::Mechanize::Shell is more than that though, it allows you to script a complete HTTP session so it can be replayed back at a later date without any interaction using WWW::Mechanize::Shell's parent perl module WWW::Mechanize - great for automatically submitting HTML forms/ POST data regularly via a cron job for example.

In this article I'll be talking about installing WWW::Mechanize::Shell, look at a typical WWW::Mechanize::Shell browsing session and look at some examples of how I use WWW::Mechanize::Shell to make things easier. Finally the article will end with a real world example - using mechshell to automate logging into FreshPorts and updating a watch list.

Installing WWW::Mechanize::Shell
As the name suggests, WWW::Mechanize::Shell is a perl module whose 'parent' is the WWW::Mechanize module written by Andy Lester (WWW::Mechanize::Shell itself is written by Max Maischein at time of original writing). WWW::Mechanize does all the work in the background - WWW::Mechanize::Shell just makes it easy to interact in a HTTP session. WWW::Mechanize::Shell and all it's dependencies can be installed from the ports tree:

CODE:

root@users /root# cd /usr/ports/www/p5-WWW-Mechanize-Shell/
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# make install
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# rehash

Getting Started Using WWW::Mechanize::Shell
Once installed, start up the WWW::Mechanize::Shell using the following you can use the following commandline:

CODE:

root@users /root# perl -MWWW::Mechanize::Shell -eshell

To make things easier though I use a CSH shell alias which aliases 'mechshell' to the command above:

CODE:

root@users /root# grep mechshell $cshrc
alias mechshell perl -MWWW::Mechanize::Shell -eshell

Examples of WWW::Mechanize Usage
I usually use WWW::Mechanize when I want to manipulate data from websites that require a stateful HTTP session - ie a browsing session where there's more than one URL you have to visit to complete the 'session'. Usually these kind of stateful sessions involve logging into the website first, then browsing to another page to obtain the data and then I have the WWW::Mechanize perl script handle the data and return any results on the commandline.

Some examples of scripts that I've use WWW::Mechanize with:

eclipse_flex_speed.pl
My ISP (Eclipse UK) used to allow you to 'flex' your internet speed from 256k up to 2Mbps. They ran an offer for a while where you could flex to the max for 3 months - unfortunately you could only flex for 12 hours at a time, which meant logging into the control panel every 12hrs, selecting the maximum speed and then submitting the form. PITA basically.

Instead I wrote eclipse_flex_speed.pl to automatically login to the Eclipse control panel, 'click' the 2Mbps radio button and then submit the form so my speed got flexed automagically. I then added the script as a cron job to autorun every 12hours, saving the haslle of doing it all manually!

aod_get.pl
The BBC website allows you to listen to streams of all BBC radio broadcasts for up to a week after they've been aired live. The problem is that the web interface you listen to the stream on in your web browser only allows you to skip 5 or 15 minutes ahead in time and doesn't allow you to go to specific times in the stream. To get around this you can obtain the URL of the real player stream and open it in a standalone real player - doing this you can go to any point in the stream easily. Trouble is finding the URL of the stream isn't that easy and involves viewing the source HTML of the web UI and copy/pasting a partial URL.

I started to write a WWW::Mechanize script to automate the 'screen scraping' of all the available feeds from the BBC Audio On Demand site and listing them on one single HTML page linking the name of the feed to the real player feed URL. As it goes though, someone else - Dave Cross - already had the same idea and wrote a great script for scraping the BBC feeds automatically. I now run this in a cronjob once a week.

torrentflux_ctl.pl
This is a script for starting and stopping all torrents under the control of the torrentflux web based bittorrent client. The script logs in as the torrent owner and then stops or starts all the torrents for that user - basically just does a GET of a URL that causes torrentflux to stop or start all torrents. Crude but effective.

Real world example - Automating the update of FreshPorts watch list
Below is a real world example usage of WWW::Mechanize::Shell - automating the procedure of updating your watch list on Freshports.org. I've included comments as '# this is a comment' to help explain what each command is doing.

CODE:

# Start up mechshell - alias for 'perl -MWWW::Mechanize::Shell -eshell':
munk@users /home/munk# mechshell

# Request the URL http://www.freshports.org/login.php.
# Note the HTTP response '(200)' is displayed underneath
# to indicate the page was fetched successfully:
(no url)>get http://www.freshports.org/login.php
Retrieving http://www.freshports.org/login.php(200)

# Use the mechshell 'dump' command to dump the contents
# of all forms found on the login page:
http://www.freshports.org/login.php>dump
POST http://www.freshports.org/login.php?origin=%2F [l]
 custom_settings=1 (hidden readonly)
 LOGIN=1 (hidden readonly)
 UserID= (text)
 Password= (password)
 submit=Login (submit)
 <NONAME>=<UNDEF> (reset)

# There's just a single form on this page:
# - the form's 'ACTION' is set to submit the form using the POST method
# to the url http://www.freshports.org/login.php?origin=%2F
# The form contains the following form fields:
# - 2 hidden fields
# - 1 text field called 'UserID'
# - 1 password field called 'Password'
# - 1 submit field called 'Login'

# Fill in the 'UserID' and 'Password' fields:
http://www.freshports.org/login.php>value UserID munk
http://www.freshports.org/login.php>value Password xxxxxx

# And then submit the form. Note we can just use the mechshell 'submit'
# command here because there is only a single form on the page. If there were
# more than one form on the page we would need to specify which button exactly to
# click:
http://www.freshports.org/login.php>submit
200

# Again note that the '200' response indicates the request was successful.
# Also note that the next mechshell prompt below has changed from
# 'http://www.freshports.org/login.php>' to just 'http://www.freshports.org/' -
# this indicates that the login script has probably redirected us to the
# freshports home page.

# Now we take a look to check that the login succeeded ok. To do this we use
# the mechshell 'content' command which effectively dumps the content of the
# returned page back at us in a pager.
# What we're looking for is the text 'Logged in as munk' which will indicate we
# logged in ok:
http://www.freshports.org/>content
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
-snip-
 <td NOWRAP>Logged in as munk <a href="http://freebsd.munk.me.uk/customize.php?origin=%2F" title="Customize your settings">Customize</a
> <a href="http://freebsd.munk.me.uk/logout.php" title="Logout of the website">Logout</a> <a href="/my-flagged-commits.php" title="Li
st of commits you have flagged">My Flagged Commits</a> 
-snip-

# Now we're logged in ok we can continue to upload the mypkg_info.txt file we
# created earlier.

# First browse to the pkg_upload.php page:
http://www.freshports.org/>get http://www.freshports.org/pkg_upload.php
Retrieving http://www.freshports.org/pkg_upload.php(200)

# Now use 'dump' to see a list of form fields on this page.
# Note that there are 2 submit buttons on this page:
http://www.freshports.org/pkg_upload.php>dump
POST http://www.freshports.org/pkg_upload.php (multipart/form-data)
 pkg_info= (file)
 staging=Staging (submit)
 wlid=5393 (option) [*5393/main*]
 replaceappend=replace (radio) [*replace/Replace list contents|append/Append to list (duplicates will be removed)]
 upload=Upload (submit)

# We need to fill out the form here. Uploading files with mechshell is as
# simple as completing the correct file type field:
http://www.freshports.org/pkg_upload.php>value pkg_info /tmp/mypkg_info.txt

# Ok, now we're ready to submit the form.
# Note that because there are 2 submit buttons on this form, we must explicitly
# tell mechshell which button it is that we want to click on - to do that we use
# the 'click' command. Just using 'submit' here would possibly click on the
# 'staging' button which is not what we want - instead we use the command
# 'click upload' to indicate we want to click on the 'upload' button:
http://www.freshports.org/pkg_upload.php>click upload
(200)

# Success! It's a good idea now to just check that this worked by browsing in
# a web browser to your watch list and checking the new items were updated ok (of
# course you can do this in mechshell if you want but I'll leave that out here!).

# Finally, the really cool bit. The mechshell 'script' command will dump out
# the perl code required to perform all of the above actions again if you copy
# them into a perl script:
http://www.freshports.org/pkg_upload.php>script
#!perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
-snip-

# Also, if you provide a filename as an argument to the 'script' command,
# mechshell will dump all the script commands to that filename:
http://www.freshports.org/pkg_upload.php>script /tmp/freshports_update.pl

# Finally, use 'quit' to exit the mechshell:
http://www.freshports.org/pkg_upload.php>quit
munk@users /home/munk#

Now all that remains is to open up /tmp/freshports_update.pl and tidy the script up so that it's more suitable for automated use via cron. For example, any 'dump' and 'content' commands can be taken out - these would only cause problems anyway if run from a non-interactive shell as used by cron.

We also need to add some code to have the script dump the contents of 'pkg_info -qoa' to a temporary file prior to uploading.

The completed 'quick and dirty' hack looks like this then:

CODE:

#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;

# FreshPorts username/pass:
my $user="munk";
my $pass="xxxxx";

# Temp location to store output from 'pkg_info -qoa':
my $mypkg_info="/tmp/freshports/mypkg_info.txt";

# prepare file containing output from: pkg_info -qoa
`pkg_info -qoa > $mypkg_info`;

# Prepare WWW::Mechanize:
my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
$agent->env_proxy();

# Login to FreshPorts:
$agent->get('http://www.freshports.org/login.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('UserID', $user); };
{ local $^W; $agent->current_form->value('Password', $pass); };
$agent->submit();

# Submit pkg_info details to FreshPorts pkg_upload page:
$agent->get('http://www.freshports.org/pkg_upload.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('pkg_info', $mypkg_info); };
$agent->click('upload');

# Remove temporary file:
`rm $mypkg_info`;

After saving the script and making the file executable, an entry can then be added to cron to have the script auto update the list of ports at freshports once a week - or however often you require it to be updated, once a week is more than enough for me. Sorted! :)

Migrating server to new hdd

nospam@example.com (munk) — Thu, 01 Feb 2007 00:56:28 +0000

Never know quite how to describe this 'migration' - it's the same old bits in the machine but I've built a new installation of FreeBSD onto a new HDD and am now putting this new 'server' (hdd!) into the old machine! Sounds pretty weak/easy maybe but it's taken me almost a month to get this done what with migrating various bits and bobs and tidying up crap along the way - together with a heavy dose of procrastination along with it all. Just managed to whittle the MySQL dbs down to about 25% of what they used to be, so many dbs and users that haven't used anything since 2003 or so!

Mind, not that it's active that much now, we're talking maybe a dozen users or so...

Anyway, still got a lot to do, the server is just about running now with the new HDD, managed to get named up and running along with apache, exim and mysql with all the data more or less migrated over now. There *will* be lots of problems with web services but that can wait for now, nothing that can't wait til tomorrow.

Installing Exim, SASLAuthd, ClamAV and SpamAssassin on FreeBSD 6.2

nospam@example.com (munk) — Wed, 17 Jan 2007 20:19:00 +0000

Introduction
This article describes the steps necessary to install and configure Exim on FreeBSD 6.2 with support for the following:

authenticated SMTP (asmtp) using SASLAuthd
spam detection and quarantine using SpamAssassin
malware detection and quarantine using ClamAV

Each of the required 'dependencies' or components will be installed and configured, Exim will be installed and configured and finally we will test to check each component is working as required.

With regards to spam and malware scanning, the system described will quarantine any files/messages that it finds classified as spam or malware. In this way the quarantined files can be checked over by the admin at a later date and various stats gathering can be done if required.

Installing and Configuring SASLAuthd
SASLAuthd is an authentication daemon that can handle authentication requests from 3rd party applications such as Exim - generally for any application that can't directly access a system password database because of permission restrictions. In this case running Exim MTA as 'root' is a potential security risk, so exim runs as the 'mailnull' user on FreeBSD. Unfortunately this means Exim can't easily read the system password database to authenticate users who want to send mail via the server, which is where SASLAuthd comes in. Any requests for authentication with Exim are passed on to the SASLAuthd daemon which will then verify whether the user credentials are valid - if so, the email is delivered, if not, it's rejected.

Install SASLAuthd from the FreeBSD ports tree:

CODE:
root@win /root# cd /usr/ports/security/cyrus-sasl2-saslauthd/
root@win /usr/ports/security/cyrus-sasl2-saslauthd# make install
...
root@win /usr/ports/security/cyrus-sasl2-saslauthd# rehash
Configure SASLAuthd to run at boot.

Edit /etc/rc.conf to include the following:

CODE:
saslauthd_enable="YES"
saslauthd_flags="-a getpwent"

Note:
SASLAuthd will run using the 'getpwent' authentication mechanism with the flag above. This method uses the passwd file directly instead of using other means like kerberos or PAM. If you require another method, check the manpage for saslauthd.
Start the SASLAuthd daemon running:

CODE:
root@win /usr/ports/security/cyrus-sasl2-saslauthd# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# ./saslauthd start
(Optional) Test the SASLAuthd daemon:

Substitute 'user' and 'pass' for the username and password of a user
account on your system:

CODE:
root@win /usr/local/etc/rc.d# testsaslauthd -u user -p pass
0: OK "Success."

Installing and Configuring SpamAssassin
SpamAssassin (SA) is one solution to the problem of spam. SA can run as a daemon (spamd) in the background and accept requests from an MTA such as Exim to check whether an email message should be classified as spam.

Spamd looks at the message and checks for various factors that make the message more or less likely to be spam and assigns the message a score based on what it finds. Spamd will then reply to the MTA, telling it the spam score that it gave that message. The MTA can then decide - based on that score - whether to accept/reject the message - or in the case of this guide whether to instead quarantine the message.

Install SA from the FreeBSD ports.

Note:
There are various installation options you can choose when installing SA which you should see when you first run 'make install' in the SA port directory. To see the options after already configuring them you can run 'make config'.

In turn, each of SA's dependencies may also have options you can configure at install time.

To write this guide I'm only using the single option 'AS_ROOT' in the SA install configuation and for the other items generally just choose the
defaults.

CODE:
root@win /root# cd /usr/ports/mail/p5-Mail-SpamAssassin/
root@win /usr/ports/mail/p5-Mail-SpamAssassin# make install

Once complete, you should see:

CODE:
*************************************************************************
*           _  _____ _____ _____ _   _ _____ ___ ___  _   _             *
*          / \|_   _|_   _| ____| \ | |_   _|_ _/ _ \| \ | |            *
*         / _ \ | |   | | |  _| |  \| | | |  | | | | |  \| |            *
*        / ___ \| |   | | | |___| |\  | | |  | | |_| | |\  |            *
*       /_/   \_\_|   |_| |_____|_| \_| |_| |___\___/|_| \_|            *
*                                                                       *
*       See /usr/local/share/doc/p5-Mail-SpamAssassin/INSTALL,          *
*       and /usr/local/share/doc/p5-Mail-SpamAssassin/UPGRADE,          *
*       or http://spamassassin.org/dist/INSTALL and                     *
*       http://spamassassin.org/dist/UPGRADE BEFORE enabling            *
*       this version of SpamAssassin for important information          *
*       regarding changes in this version.                              *
*************************************************************************

It's a good idea to read the files listed in the banner above. SA has a large number of options that can be configured; a good place to start configuring options on FreeBSD is in /usr/local/etc/mail/spamassassin/.
Configure SA to run at boot.

Edit /etc/rc.conf to include the following:

CODE:
spamd_enable="YES"
Start SA spamd.

We can now go on to actually start spamd running as a daemon and verify spamd started ok:

CODE:
root@win /root# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# rehash
root@win /usr/local/etc/rc.d# ./sa
sa-spamd* saslauthd*
root@win /usr/local/etc/rc.d# ./sa-spamd start
Starting spamd.
munk@win /usr/local/etc/rc.d# ./sa-spamd status
spamd is running as pid 754.

This tells us spamd is running ok in the background.

Installing and Configuring ClamAV
ClamAV is an anti-virus suite and includes a daemon clamd (runs in the background to check for requests to test for virii), another daemon freshclam (updates the virus definition database) and a couple of clients to run on the commandline if you need them for local virus scanning.

Exim will send requests to the clamd server in much the same was as spamd does - if clamd classifies a message as containing a virus, Exim will reject delivery of the message and instaed quarantine it.

Install ClamAV from the FreeBSD ports tree:

CODE:
root@win /root# cd /usr/ports/security/clamav
root@win /usr/ports/security/clamav# make install
Configure ClamAV to start at boot time.

Edit /etc/rc.conf to include:

CODE:
clamav_clamd_enable="YES"
clamav_freshclam_enable="YES"
Configure clamd.

Edit /usr/local/etc/clamd.conf to include the following:

CODE:
LogFile /var/log/clamav/clamd.log
PidFile /var/run/clamav/clamd.pid
DatabaseDirectory /var/db/clamav
LocalSocket /var/run/clamav/clamd
FixStaleSocket
User clamav
AllowSupplementaryGroups
ScanMail
ScanArchive
Start clamd and freshclam.

CODE:
root@win /root# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# ./clamav-clamd start
Starting clamav_clamd.
root@win /usr/local/etc/rc.d# ./clamav-freshclam start
Starting clamav_freshclam.

Note:
You may see the following message on first running clamd:

CODE:
LibClamAV Warning: **************************************************
LibClamAV Warning: *** The virus database is older than 7 days. ***
LibClamAV Warning: *** Please update it IMMEDIATELY! ***
LibClamAV Warning: **************************************************

As long as you're running freshclam, you can safely ignore this message. Freshclam should update your definitions automatically. Be sure to configure freshclam to update the virus definitions regularly.

ClamAV should be configured now and ready to accept request to check for malware from the Exim MTA.

We can now move on finally to install and configure Exim.

Installing and Configuring Exim
Exim configuration can be very complicated. This guide will only deal with the configuration of Exim so it accepts mail on a domain 'mail.example.com', scans the mail for malware/spam - quarantining anything it finds as malware/spam and accepts authentication requests correctly.

Important:
Ensure your mail server's DNS is configured correctly and preferably has a reverse DNS record (rDNS) set up. Many mail servers will not deliver mail correctly to/from your mail server without rDNS.

Install Exim from the FreeBSD ports tree:

CODE:
[12:10:57] root@win /root# cd /usr/ports/mail/exim
[12:12:30] root@win /usr/ports/mail/exim# make -DWITH_CONTENT_SCAN -DWITH_SASLAUTHD install
Stop the Sendmail daemon if it's already running:

CODE:
root@win /root# cd /etc/rc.d
root@win /etc/rc.d# ./sendmail stop
Configure Exim to run at boot time.

Edit /etc/rc.conf to include:

CODE:
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
exim_enable="YES"

This has the effect of disabling sendmail at boot time - the default FreeBSD MTA - and running Exim instead.
Configure mailer.conf to use Exim as the default MTA.

Edit /etc/mail/mailer.conf to read:

CODE:
sendmail          /usr/local/sbin/exim
send-mail         /usr/local/sbin/exim
mailq             /usr/local/sbin/exim -bp
newaliases        /usr/bin/true

This will allow any FreeBSD base system mail related commands to use Exim instead of Sendmail.

Configuring Exim
We now move on to configuring Exim.

Set the primary hostname.

Edit /usr/local/etc/exim/configure.

Find and edit the 'primary_hostname' line for your domain:

CODE:
primary_hostname = example.com

This configures Exim to accept mail primarily for the 'example.com' domain - ie foobar@example.com.
Find and edit the following lines to read:

CODE:
av_scanner = clamd:/var/run/clamav/clamd
spamd_address = 127.0.0.1 783
Configure the malware and spam Access Control Lists (ACLs).

How malware/spam checking works in this system:
We add a check in the acl_check_data ACL for spam and malware. Exim will request each email is checked for spam/malware by the relevant daemon - spamd for spam, clamd for malware. If the message is classified as spam/malware by the relevant daemons, Exim will add a header to the message 'X-Quarantine-Me-Spam' (similar for malware).

Later on when it comes to actually delivering (termed 'routing' in Exim terminology), we add two routers to test for the existence of the headers that are added in the acl_check_data ACL if a message is found to be spam/malware. If the headers are found by the malware/spam routers, the message is not delivered but instead copied to a quarantine location on disk.

This quarantine location can then be checked later by an admin to check if anything is amiss - ie regular non spam/malware mail that should really have been delivered.

Once you're satisfied the configuration is working as it should - ie after a few months of operation - and not finding false positives, you can change the malware/spam acl checks to just deny instead of adding the quarantine headers. Having said that, I still opt to just quarantine malware/spam and remove it at a later date.

On to configuring the data ACL:

Modify the acl_check_data ACL to read/include:

CODE:
acl_check_data:

  # Deny if the message contains a virus. Before enabling this check, you
  # must install a virus scanner and set the av_scanner option above.
  #
  # defer_ok - pass this message if scanner is down etc:
  warn  message         = X-Quarantine-Me-Malware: $malware_name
        log_message     = malware: $malware_name
        demime          = *
        malware         = */defer_ok

  # Add headers to a message if it is judged to be spam. Before enabling this,
  # you must install SpamAssassin. You may also need to set the spamd_address
  # option above.
  #
  warn    message       = X-Quarantine-Me-Spam: SA score $spam_score\n\
                          X-SA-Report: $spam_report
          log_message   = Spam score $spam_score > 5
          spam          = spamd/defer_ok
          condition     = ${if >{$spam_score_int}{50}{1}{0}}

  # Accept the message.

  accept

At the top of the routers section, modify to read/include:

CODE:
begin routers

check_malware:
  driver                = redirect
  condition             = ${if def:h_X-Quarantine-Me-Malware: {1}{0}}
  headers_add           = X-Quarantined-Malware: $h_X-Quarantine-Me-Malware:
  headers_remove        = X-Quarantine-Me-Malware
  data                  = /var/quarantine/malware/malware.$tod_logfile
  file_transport        = address_file

check_spam:
  driver                = redirect
  condition             = ${if def:h_X-Quarantine-Me-Spam: {1}{0}}
  headers_add           = X-Quarantined-Spam: $h_X-Quarantine-Me-Spam:
  headers_remove        = X-Quarantine-Me-Spam
  data                  = /var/quarantine/spam/spam.$tod_logfile
  file_transport        = address_file

no_more

Modify the authenticators section to read:

CODE:
begin authenticators
plain:
driver = plaintext
public_name = PLAIN
server_condition = ${if saslauthd{{$2}{$3}}{1}{0}}

login:
driver = plaintext
public_name = LOGIN
server_prompts = "Username:: : Password::"
server_condition = ${if saslauthd{{$1}{$2}}{1}{0}}

Save the /usr/local/etc/exim/configuration file.
Create the quarantine directories and change ownership to mailnull:mail:

CODE:
root@win /root# mkdir -p /var/quarantine/{malware,spam}
root@win /root# chown mailnull:mail /var/quarantine/{malware,spam}
Restart Exim to suck in the new config options:

CODE:
root@win /root# /usr/local/etc/rc.d/exim restart
Stopping exim.
Starting exim.

Exim should now be set to check for malware/spam and to authenticate users.

Testing Exim configuration
Finally we can move on to test that our config works correctly for spam/malware checking and for authenticating users.

Testing Exim's malware/spam scanning.

The easiest option is to send an email to your mailserver with specially crafted malware/spam signatures included in the body of the message. When spamd/clamd see these signature strings in the body of the messages, they should classify the message as spam/malware and Exim in turn will quarantine the messages.

The official EICAR malware/virus testing signature is as follows:

CODE:
X5O%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILEspamcH+H*

See here for the official string:

http://www.eicar.org/anti_virus_test_file.htm

The official GTUBE spam testing signature is as follows:

CODE:
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

See here for the official string:

http://spamassassin.apache.org/gtube/

Note:
Another option for testing spam/malware scanning is to run exim from the commandline using the command 'exim -bh 127.0.0.1'. This will run an SMTP session from the commandline (think telnet) and allow you to inject your own specially crafted message using the signatures above. This requires you enter a valid SMTP session, something like:

CODE:
HELO example.com
MAIL FROM:foo@example.com
RCPT TO:foo@example.com
DATA
X5O%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILEspamcH+H*
.

This would simulate the injection of a mail message with a virus in it and in 'exim -bh' mode you can see a lot of useful debugging info to verify everything works ok.
Testing Exim's Authentication configuration.

We can now test that ASMTP is working. For this you can either run exim in one of it's many excellent debugging modes or you can simply configure a remote email client to use ASMTP. This guide will use the commandline to test ASMTP.

Important:
Before attempting this method please read the exim documentation on how ASMTP works. The following assumes you have read and understood that text.

First create a simple perl script called 'encode' in /usr/local/etc/exim/ and make sure it is executable:

CODE:
root@win /usr/local/etc/exim# cat encode
#!/usr/bin/perl
use MIME::Base64;
printf ("%s", encode_base64(eval ""$ARGV[0]""));
root@darkstar /usr/local/etc/exim# chmod +x encode
root@darkstar /usr/local/etc/exim# ls -al encode
-rwxr-xr-x 1 root wheel 85 Apr 23 12:25 encode

Now decide which user account on your server you wish to test ASMTP with. It must be an account you know the password for obviously. I created an account called 'dummy' and set the password to 'dummy' as well - if you do this remember to remove the account or disable it as soon as you've finished testing.

Encode the user:password pair into base64 MIME using the 'encode' script we created above:

CODE:
root@darkstar /usr/local/etc/exim# ./encode "\0dummy\0dummy"
AGR1bW15AGR1bW15

Now enter into Exim's fake SMTP session command-line mode and just for good measure do it in authentication debug mode as well:

CODE:
root@win /root# exim -d+auth -bh 127.0.0.1
Exim version 4.66 (FreeBSD 6.1) uid=0 gid=0 pid=3056 D=fbb95cfd
Probably Berkeley DB version 1.8x (native mode)
Support for: crypteq iconv() IPv6 use_setclassresources PAM Perl OpenSSL Content_Scanning Old_Demime
Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch nis nis0 passwd
Authenticators: cram_md5 plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Size of off_t: 8
changed uid/gid: forcing real = effective
 uid=0 gid=0 pid=3056
 auxiliary group list: 0
seeking password data for user "mailnull": using cached result
getpwnam() succeeded uid=26 gid=26
seeking password data for user "root": cache not available
getpwnam() succeeded uid=0 gid=0
configuration file is /usr/local/etc/exim/configure
log selectors = 00000ffc 00089001
trusted user
admin user
changed uid/gid: privilege not needed
 uid=26 gid=6 pid=3056
 auxiliary group list: 6 6
seeking password data for user "mailnull": cache not available
getpwnam() succeeded uid=26 gid=26
originator: uid=0 gid=0 login=root name=Charlie Root
sender address = root@win.munk.me.uk
sender_fullhost = [127.0.0.1]
sender_rcvhost = [127.0.0.1]

**** SMTP testing session as if from host 127.0.0.1
**** but without any ident (RFC 1413) callback.
**** This is not for real!

host in hosts_connection_nolog? no (option unset)
LOG: smtp_connection MAIN
 SMTP connection from [127.0.0.1]
host in host_lookup? yes (matched "*")
looking up host name for 127.0.0.1
DNS lookup of 1.0.0.127.in-addr.arpa (PTR) succeeded
IP address lookup yielded localhost.munk.me.uk
gethostbyname2 looked up these IP addresses:
 name=localhost.munk.me.uk address=::1
 name=localhost.munk.me.uk address=127.0.0.1
checking addresses for localhost.munk.me.uk
 ::1
 127.0.0.1 OK
sender_fullhost = localhost.munk.me.uk [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1])
set_process_info: 3056 handling incoming connection from localhost.munk.me.uk [127.0.0.1]
host in host_reject_connection? no (option unset)
host in sender_unqualified_hosts? no (option unset)
host in recipient_unqualified_hosts? no (option unset)
host in helo_verify_hosts? no (option unset)
host in helo_try_verify_hosts? no (option unset)
host in helo_accept_junk_hosts? no (option unset)
SMTP>> 220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
smtp_setup_msg entered

When you get to this point you are ready to start an SMTP 'conversation' with Exim. First introduce yourself to Exim using the SMTP 'EHLO localhost' command:

CODE:
EHLO localhost
SMTP<< EHLO localhost
sender_fullhost = localhost.munk.me.uk (localhost) [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1] helo=localhost)
set_process_info: 3103 handling incoming connection from localhost.munk.me.uk (localhost) [127.0.0.1]
host in pipelining_advertise_hosts? yes (matched "*")
host in auth_advertise_hosts? yes (matched "*")
host in tls_advertise_hosts? no (option unset)
250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
SMTP>> 250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP

In response to your 'EHLO localhost' command, Exim returns more debug information but most importantly for us it also indicates what authentication options it offers in this line:

CODE:
250-AUTH PLAIN LOGIN

This indicates that currently acceptable AUTH methods are PLAIN and LOGIN.

We can then test the PLAIN login method using the "\0dummy\0dummy" user:password pair we encoded above:

CODE:
AUTH PLAIN AGR1bW15AGR1bW15
SMTP<< AUTH PLAIN AGR1bW15AGR1bW15
Running pwcheck authentication for user "dummy"
pwcheck: success (NULL)
plain authenticator:
 $1 =
 $2 = dummy
 $3 = dummy
expanded string: 1
SMTP>> 235 Authentication succeeded
235 Authentication succeeded

This indicates that authentication for 'dummy:dummy' would succeed and mail would be relayed (pending further conditional checks by Exim).

So we now have a working Exim with support for spam/malware checking and authentication over SMTP.

Cursor Artifacts Using nv nVidia drivers on X

nospam@example.com (munk) — Mon, 15 Jan 2007 12:26:28 +0000

Just come across an annoying problem whilst using a GeForce 6200 in X - in Firefox, scrolling around inside form fields or the URL bar with the arrow keys results in a number of cursor artifacts left behind making reading or editing anything in those areas impossible. Switching focus away and then back to the Firefox window removes the artifacts, but this isn't very practical.

After some digging around to confirm the bug is with the 'nv' driver (the problem disappears when running with just the vesa driver), I scanned the bugs listed for the nv driver and thankfully found the bug listed already here.

The bug report also thankfully had a simple workaround to the problem - adding:

CODE:

Option "XaaNoSolidFillRect"

in the card's device section. No idea what the drawbacks are to using this option, searching for the term "XaaNoSolidFillRect" just comes up with a list of other nv bug reports citing the above as a solution. From those results, the same problem is described in more detail also at this URL by another nv user with an ascii art representation of the problem (woot!):

http://lists.debian.org/debian-x/2006/01/msg00056.html

FreeBSD installation problems with Maxtor DiamondMax 10 250 Gb HDD 6L250R0

nospam@example.com (munk) — Fri, 05 Jan 2007 14:30:21 +0000

As I mentioned in another post, I recently got a new Maxtor DiamondMax 10 250Gb HDD - model is 6L250R0 - and decided to install FreeBSD 6.1 on it. I downloaded and burnt a copy of the distribution to disc, booted from the installer OK and everything went well until it got to the installation commit part where I received a number of errors like this:

CODE:

ad0: WARNING - READ_DMA UDMA ICRC error (retrying request) LBA=0
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=84<ICRC,ABORTED> LBA=0

The short version of the solution is to either use an 80 conductor IDE cable to allow correct Ultra DMA transfers OR run the FreeBSD installation in safe mode. The first option is
much prefered, even if only to increase the transfer speed used by the HDD. Installing in safe mode might work ok, but the underlying issues with UDMA may still persist after the installation when the operating system is in use, which is obviously not a good thing. If possible go for upgrading the IDE cable to 80 conductor.

More details of the problem are detailed below:

The problem occurs at the point where the installer attempts to write the disk labels. The disk appears to be formatted ok and the FreeBSD slices (DOS 'partitions') are created ok, the errors occur when the FreeBSD partitions inside each slice are created (ad0s1a-h).

(Incidentally I found that I could only assign a maximum of 7 partitions per slice which seems to contradict what I read somewhere else about this saying that you could create up to 8 partitions per slice. Perhaps this is to do with swap partition creation though, I'm not sure, the 7 does include one swap partition - ie not 7 regular parts plus the swap...)

Anyway back to the problem. After a quick search I found this post which almost exactly describes my problem along with a much appreciated reply - qutoe from that post:

QUOTE:

Finally I got things working!

First I tried several other IDE cables, which did not help.

Then I tried Carl's advice and bought a 80-connector cable. And amazingly
enough... It worked!

From what I can gather the problem seems to be that the FreeBSD installer doesn't detect the DMA mode correctly and tries to use the faster UDMA mode (Ultra DMA) regardless of whether a compatible 80 conductor IDE cable is in use or not (an 80 conductor cable is required to make use of the extra speed of UDMA (see here for a good description of 80 conductor IDE cables) and attempting to use UDMA without an 80 conductor cable causes problems *from what little I understand of the thing*). Why this is missing on FreeBSD I don't know - it seems to be detected OK on Windows XP and from what I read, OpenBSD works fine around this problem too.

Anyway, I ordered an 80 conductor IDE cable from ebay so hopefully that will make installation easier and generally increase speed as well for the HDD. Of course by sod's law, as soon as I'd done this I found a way to get around the problem...

Out of curiousity I decided to try and use the FreeBSD installer's 'Safe Mode' to see if it would fall back to using a safer DMA mode. I had to use a standard PS/2 keyboard instead of my USB one - presumably because USB support isn't included in safe mode - but the installer actually managed to write to the disk properly this time which is good.

The caveat, as mentioned above, is that the problems may still persist after the installation, I've not tested this out as yet so don't know but would be surprised if the problems magically dissappeared without using an 80 conductor cable. End of the day, make sure to use an 80 conductor IDE cable if you run across this kind of problem on FreeBSD.

Moving munk.me.uk to a new HDD - filesystem layout

nospam@example.com (munk) — Fri, 05 Jan 2007 14:25:00 +0000

Well after having been on FreeBSD 4.x for as long as I can remember (4years now I guess), it's finally time to move to FreeBSD 6.x! For ages I've been saying as soon as I get a larger HDD I'd upgrade the system and santa kindly delivered a shiny new 250Gb Maxtor DiamondMax 10, so it's time to sort it out and make the move.

Funny now I think about it, for me upgrading to a larger capacity HDD has always been a fairly rare occasion. Back in 2000 or so I remember moving from a measily 1.6Gb to a whopping 20Gb HDD and thinking that was way too much. In a way it's a good thing to not have a lot of space - it makes you more tidy and less likely to spam crap all over the filesystem. Of course on the other hand not having a lot of space also sucks if you want to store a lot of stuff (duh). This is kind of how it's been with my FreeBSD server for the last 4yrs or so - I've only had a 40Gb drive and in that time I've hosted over 100 users at the same time, dozens of domains, ran a load of services, and never really had a lot to complain about re lack of space. The main reason for the change I guess is the increased use of broadband/bittorrent which I really need more space to save files to disk for.

I'm gonna document the filesystem layout of the new HDD here, no doubt it'll only be me that ever reads this again (and you if you're reading this heh :o).

File System Layout
I spent quite a while thinking about how best to partition/slice up the new 250Gb disk, mainly because my server is running 24/7 and various applications really do kane the filesystem when accessing data (ie apache and mysql for two). I want to try and partition the disk so that it's more efficient for apache and mysql to read/write from/to the disks.

The other deliberation I went through whilst thinking about the file system layout was where to mount partitions for backups, music, videos and windows application installers. Of course this could be anywhere really - /backups /music /videos /windows for example - but I don't really like spamming the root level filesystem with lots of folders that only add clutter.

I had a quick look at the Filesystem Hierarchy Standard which is a standard aimed at encouraging clean and consistent filesystem layouts on Unix type OSs and eventually ended up with the following layout:

CODE:

G=Gigabytes

fdisk:
======

Name        Size                Partition Type
====        ====                ==============
ad1s1        59G                freebsd
ad1s2        89G                freebsd
ad1s3        49G                freebsd
ad1s4        33G                freebsd

disklabel:
==========

Partition    Mount              Size (approx.)
=========    =====              ==============
ad1s1:
------

ad1s1a        /                 0.5G

ad1s1b        swap              2G

ad1s1d        /tmp              1G

ad1s1e        /var/db/mysql     20G

ad1s1f        /var/www          10G

ad1s1g        /var/             10G

ad1s1h        /usr              20G

ad1s2:
------

ad1s2d        /home             40G

ad1s2e        /var/backups      50G

ad1s3:
------

ad1s3d        /var/media        50G

ad1s4:
------

ad2s4d        /var/win32        35G

Block Brute Force Attacks Against sshd and proftpd Using blockhosts

nospam@example.com (munk) — Sat, 30 Dec 2006 11:52:15 +0000

For a long time now I've had a lot of problems with brute force attacks against sshd and proftpd - attacks where a host will attempt to login with a dictionary of common usernames and passwords, trying each one until they find a combination that works. Apart from being a security issue, this uses up a lot of bandwidth so it's worth taking some measures to block these kind of attacks.

Both sshd and ftpd services have their own individual means for blocking individual connections, but unfortunately neither have an inbuilt method for detecting brute force attacks - counting how many failed login attempts are made from each individual IP address and then blocking that IP address if the number of failed login attempts is more than a certain number. This is where a 3rd party utility is required.

There are a few utilities that can mitigate brute force attacks on services. For a while now I've used DenyHosts successfully to block sshd brute force attacks. DenyHosts works by constantly monitoring sshd logfiles and keeping track of how many failed logins have occured per IP address over time. If the number of failed logins reaches a certain threshold, DenyHosts adds an entry in /etc/hosts.allow that effectively blocks the IP address, stopping that host from connecting to the sshd service any more.

DenyHosts is great, but unfortunately it's aimed only at blocking sshd brute force attacks and I need to protect the ftpd service as well as just sshd - and in future maybe adapt to block other services. With this in mind I decided to move to using a very similar script called BlockHosts (the documentation for BlockHosts actually mentions that it was inspired by DenyHosts). BlockHosts can scan a list of service logfiles in one go instead of just a single logfile as with DenyHosts, so is ideal for monitoring a number of different services for brute force attacks.

The following describes how to install and configure BlockHosts on FreeBSD so it's executed every time the sshd or proftpd services are accessed using TCP_WRAPPERS - ie modifying /etc/hosts.allow so the blockhosts script is run each time sshd or proftpd are accessed. The BlockHosts script will then check if this current connection attempt is part of a brute force attack and if so, add a blocking rule to /etc/hosts.allow to deny further access.

Installation of BlockHosts

Download blockhosts from the download page, extract the distribution (note please check the download link for the latest version, the version below was latest at time of writing):

CODE:
root@users /home/munk/bin/python/blockhosts# wget http://www.aczoom.com/tools/blockhosts/BlockHosts-1.0.5.tar.gz
root@users /home/munk/bin/python/blockhosts# tar zxvf BlockHosts-1.0.5.tar.gz
BlockHosts-1.0.5/
BlockHosts-1.0.5/Makefile
...
Change to BlockHosts directory:

CODE:
root@users /home/munk/bin/python/blockhosts# cd BlockHosts-1.0.5
Edit and save blockhosts.py to read:

CODE:
CONFIG_FILE = "/usr/local/etc/blockhosts.cfg"
...
"LOGFILES": ( "/var/log/auth.log", ),

Note: may seem a bit odd editing the blockhosts.py script before it's installed - the reason for this is that the installation locations used by setup.py below are taken from blockhosts.py, so by modifying blockhosts.py like this we get the config file installed into /usr/local/etc/ (FreeBSD default for 3rd party software) instead of into /etc (default for linux 3rd party software).
Install blockhosts:

CODE:
root@users /home/munk/bin/python/blockhosts/BlockHosts-1.0.5# python setup.py -v install
running install
running build
running build_scripts
creating build
creating build/scripts-2.4
copying and adjusting blockhosts.py -> build/scripts-2.4
changing mode of build/scripts-2.4/blockhosts.py from 644 to 755
running install_scripts
copying build/scripts-2.4/blockhosts.py -> /usr/local/bin
changing mode of /usr/local/bin/blockhosts.py to 755
running install_data
copying blockhosts.cfg -> /usr/local/etc

This installs the blockhosts.py script into /usr/local/bin and the config file blockhosts.cfg into /usr/local/etc. Make sure to run 'rehash' to reread the binary paths again so blockhosts.py will run from anywhere:

CODE:
root@users /home/munk/bin/python/blockhosts/BlockHosts-1.0.5# rehash
Edit and save the /usr/local/etc/blockhosts.cfg file so it reads:

CODE:
LOGFILES = [ "/var/log/auth.log", "/var/log/ftp.log" ]

Important:
Add the logfiles you want blockhosts to monitor for brute force attacks here. /var/log/auth.log is standard for sshd, /var/log/ftp.log is maybe not standard for all ftpd, this is just what I have setup here.

At this point it's best to read through the documentation for blockhosts completely - the README, INSTALL and the blockhosts.py script itself. The following section is pretty much copy/pasted from what's mentioned in there.
Edit and save /etc/hosts.allow to include the section that blockhosts.py will modify. Make sure you allow your own IP blocks first and any trusted IPs so they don't get blocked accidentally:

GOTCHA LOOKOUT!
One gotcha to watch out for in this is the line:

CODE:
ALL : ALL : allow

You MUST remove this line - replace it with your IP block instead so you don't get locked out from your own address range. If this line isn't removed/commented out, anything below it just isn't read/executed and blockhosts won't work.

This is how my /etc/hosts.allow looks:

CODE:
#######################################################################
# blockhosts
#######################################################################
# ----
# see "man 5 hosts_access" for details of the format of IP addresses,
#services, allow/deny options. Also see "man hosts_options"
#order of lines in this file is important, first matched IP address line
#is rule applied by hosts_access
#
# permanent whitelist addresses - these should always be allowed access

ALL : 213.152.51.192/255.255.255.248 : allow
# ALL: 127.0.0.1  : allow
# ALL: 192.168.0. : allow

# permanent blacklist addresses - these should always be denied access

# ALL: 10.  : deny
# ALL: 192. : deny
# ALL: 172. : deny

# ----------------------------------------
# next section is the blockhosts section - it will add/delete entries in
# between the two marker lines (#---- BlockHosts Additions)

#---- BlockHosts Additions
#---- BlockHosts Additions

# ----------------------------------------
# finally, the command to execute the blockhosts script, based on
# connection to particular service or services, for example, for
# sshd and proftpd - if using vsftpd, pure-ftpd, be sure to use those
# words instead:

sshd, proftpd: ALL: spawn (/usr/local/bin/blockhosts.py --verbose --echo "%c-%s" >> /var/log/blockhosts.log 2>&1 )& : allow

# remove:   >> /var/log/blockhosts.log 2>&1     if logging to
# blockhosts.log is not needed - it will still log to syslog (minimally)
# see examples below
# --
# See "man hosts.allow" for info on %c and %s identifiers
#----
# for non-verbose, with identification, to syslog only (/var/log/messages):
#sshd, proftpd, in.proftpd: ALL: spawn /usr/bin/blockhosts.py --echo "%c-%s" & : allow
#----
# minimal logging, to syslog (usually goes to /var/log/messages):
#sshd, proftpd, in.proftpd: ALL: spawn /usr/bin/blockhosts.py & : allow
#----
# To test hosts.allow, and to find out exact names of SSH/FTP services,
# add this line to the beginning of hosts.allow, use ssh/ftp to connect
# to your server, and then look at the log (/var/log/messages or
# blockhosts.log) to see the name of the invoked service.
# IMPORTANT: after your test is done, remove this line from hosts.allow!
# Otherwise everyone will always have access.
#ALL : ALL: spawn (/usr/bin/blockhosts.py --verbose --echo "%c-%s" >> /var/log/blockhosts.log 2>&1 )& : allow
#######################################################################
# blockhosts
#######################################################################

Important Note for ProFTPD users:
The following sections describes the configuration needed when using proftpd via inetd. If you are using ProFTPD in standalone mode, you need to use the proftpd mod_wrap/mod_wrap_file functionality to have proftpd read and honour the TCP_WRAPPERS//etc/hosts.allow file(s) when denying/allowing hosts. Additionally you need to specifiy the configure flag --enable-wrapper-options when building proftpd. For a heavily used server, this might be worth doing but personally I don't get that many connections that I need to worry about inetd being overloaded so I can just go down the (easier to configure for blockhosts) inetd path.
Ensure proftpd is configured to run correctly via inetd.

Edit and save /usr/local/etc/proftpd.conf to read:

CODE:
ServerType inetd

Important: remember to delete or rename /usr/local/etc/rc.d/proftpd.sh so it's not run at boot time - the proftpd daemon doesn't need to be started at boot if you're using inetd, inetd handles all the proftpd connections, see below:

Edit and save /etc/inetd.conf to read:

CODE:
ftp stream tcp nowait root /usr/local/sbin/in.proftpd proftpd

then restart inetd:

CODE:
root@users /usr/local/etc# kill -HUP `cat /var/run/inetd.pid `

This forces inetd to restart, rereading the config file changes made to /etc/inetd.conf. ftp connections will now be handled by proftpd via inetd.

We're now ready to run blockhosts.py for the first time. BlockHosts will parse each logfile mentioned in blockhosts.cfg and check for any brute force attacks and if it finds any, blocks will be added to the /etc/hosts.allow file.

Note: This initial check does not take into account the period over which failed logins took place, so any IP that has more than the default 7 failed login entries will look like a brute force attacker. However, the ban BlockHosts adds will only last for the default 12 hours so this shouldn't cause a huge issue - just be aware of this and check the IPs that are added on the first run.

For the very first time it's a good idea to try a 'dry run' just to see what blockhosts finds and what it'd do, without actually doing anything to the /etc/hosts.allow file. To do this, run blockhosts with the '--dry-run' flag:

CODE:

root@users /usr/local/etc# /usr/local/bin/blockhosts.py --verbose --dry-run
blockhosts 1.0.5 started: 2006-12-30 14:15:30
... will discard all host entries older than  2006-12-30 02:15
... load blockfile: /etc/hosts.allow
... found both markers, count of hosts being watched: 0
  Warning: no offset found, will read from beginning in logfile: /var/log/auth.log
... securelog, loading file, offset: /var/log/auth.log 0
  Warning: no offset found, will read from beginning in logfile: /var/log/ftp.log
... securelog, loading file, offset: /var/log/ftp.log 0
... updates: counts: hosts to block: 9; hosts being watched: 21
#---- BlockHosts Additions
ALL:  203.88.192.225 : deny
ALL:    200.71.192.7 : deny
ALL:  212.227.81.146 : deny
ALL:    218.25.62.75 : deny
ALL:  200.46.108.164 : deny
ALL:    201.57.163.2 : deny
ALL:  205.129.191.11 : deny
ALL:    200.68.51.91 : deny
ALL:    82.38.68.217 : deny

#bh: ip:   85.184.10.200 :   1 : 2006-12-30-14-15
#bh: ip:  84.158.231.209 :   1 : 2006-12-30-14-15
#bh: ip:    82.38.68.217 :  11 : 2006-12-30-14-15
#bh: ip:    82.153.28.16 :   2 : 2006-12-30-14-15
#bh: ip:   67.113.225.66 :   1 : 2006-12-30-14-15
#bh: ip:   59.108.34.228 :   2 : 2006-12-30-14-15
#bh: ip:  222.68.192.132 :   2 : 2006-12-30-14-15
#bh: ip:    218.25.62.75 :  20 : 2006-12-30-14-15
#bh: ip:  217.83.162.157 :   1 : 2006-12-30-14-15
#bh: ip:  212.227.81.146 : 29499 : 2006-12-30-14-15
#bh: ip:   210.1.132.178 :   4 : 2006-12-30-14-15
#bh: ip:  205.129.191.11 :  20 : 2006-12-30-14-15
#bh: ip:   204.141.87.14 :   3 : 2006-12-30-14-15
#bh: ip:  203.88.192.225 : 448 : 2006-12-30-14-15
#bh: ip:  202.108.40.109 :   1 : 2006-12-30-14-15
#bh: ip:    201.57.163.2 : 2867 : 2006-12-30-14-15
#bh: ip:    200.71.192.7 : 761 : 2006-12-30-14-15
#bh: ip:    200.68.51.91 :  10 : 2006-12-30-14-15
#bh: ip:  200.46.108.164 : 170 : 2006-12-30-14-15
#bh: ip:  200.105.255.90 :   7 : 2006-12-30-14-15
#bh: ip:  152.104.125.14 :   3 : 2006-12-30-14-15

From this you can see nicely what blockhosts makes of the service logfiles and the addresses that have tried to connect unsuccessfully. On my host, as you can see above, there are a few that are obviously dodgy (I would only expect a max of maybe 8 connections per ip per month, so clearly 29,499 connections is just wrong!).

Once you're happy that the output is correct, run blockhosts again without the '--dry-run' flag and the /etc/hosts.allow file will be modified. Also from now on the logfiles will only be read from the last recorded offset which saves a lot of time if your logfiles are very big.

Big thanks to the BlockHosts author Avinash Chopde !

Portupgrade fails to upgrade dependencies

nospam@example.com (munk) — Sun, 24 Dec 2006 15:01:00 +0000

When using portupgrade to upgrade ports recursively, occasionally you get a problem where portupgrade fails to upgrade a dependency of a port that's being upgraded. This seems to happen most often with perl ports - p5-* ports - probably because perl packages/ports use the most dependencies of all with being modular in design anyway.

An example is when I just went to run the weekly 'portupgrade -arR' and whilst upgrading p5-PathTools-3.21, portupgrade found that another port - p5-Scalar-List-Utils-1.18 - also needed upgrading. Unfortunately the upgrade of that port failed with the following error:

CODE:

===>  Checking if lang/p5-Scalar-List-Utils already installed
===>   p5-Scalar-List-Utils-1.18,1 is already installed
      You may wish to ``make deinstall'' and install this port again
      by ``make reinstall'' to upgrade it properly.
      If you really wish to overwrite the old port of lang/p5-Scalar-List-Utils
      without deleting it first, set the variable "FORCE_PKG_REGISTER"
      in your environment or the "make install" command line.
*** Error code 1

Stop in /home/munk/ports/lang/p5-Scalar-List-Utils.
*** Error code 1

Stop in /home/munk/ports/devel/p5-PathTools.

The problem is that any already installed dependencies - regardless of whether they need upgrading or not - are seen by portupgrade as being installed already and so it refuses to upgrade. Hence if the port you're trying to upgrade has a dependency that *also* needs upgrading, this will fail. Adding '-f' onto the portupgrade line makes no difference.

One solution is to set the environment variable 'FORCE_PKG_REGISTER':

CODE:

setenv FORCE_PKG_REGISTER 1

and then run the portupgrade command again. The installed status of the dependencies are then effectively ignored and the port dependencies are forcibly installed. Not sure why portupgrade doesn't upgrade any dependencies automatically - I would have thought any dependencies of a port would be automatically checked for upgrades and upgraded if necessary. Maybe I'm missing something.

Let root see all files with locate

nospam@example.com (munk) — Sat, 18 Nov 2006 10:38:40 +0000

The locate utility on linux was one of the first tools I hit when I made the move to FreeBSD a few years back - knowing where files are is half the battle when you're trying to configure things and find documentation on how to do it. The trouble with locate though as jdarnold mentions in his article 'Locate This!' is that if you build the locate database as 'root', you end up exposing everything to any user that runs the locate command. The other problem he mentions is the locate db is only updated weekly on FreeBSD by default via the periodic system which isn't really enough if you use your system regularly.

I remember thinking along the same lines a while back and after reading through the man pages the solution I found was to create two separate databases - one for root and one for regular users. The 'regular' db is updated on a weekly basis as per the default on FreeBSD via periodic, whereas the other 'root' locate db is built daily in a crontab so I can get the latest up to date details on which files are where.

To get the root db built first you need to create a crontab entry - i put this in /etc/crontab:

CODE:

39 2 * * * root env -i LOCATE_CONFIG=/root/locate/conf/locate.rc /usr/libexec/locate.updatedb > /dev/null 2>&1

This tells the locate.updatedb script to use a separate configuration file - /root/locate/conf/locate.rc - for building root's locate db. The content of /root/locate/conf/locate.rc look like this:

CODE:

FCODES="/root/locate/db/locate.database.root"

which indicates that this db should be built in /root/locate/db/locate.database.root instead of the default locate in /var/db/locate.database. You can safely run the command as root on the commandline to initialize your new db:

CODE:

root@users /root# env -i LOCATE_CONFIG=/root/locate/conf/locate.rc /usr/libexec/locate.updatedb

Once the database is built you can move on to test the new db works ok:

CODE:

root@users /root# locate -d /root/locate/db/locate.database.root .cshrc.root
/root/.cshrc.root

This file is only readable by root, so it seems to work ok.

To make things easier, add a shell alias in root's .cshrc file aliasing 'locate' to the command 'locate -d /root/locate/db/locate.database.root':

CODE:

root@users /root# grep locate $cshrc
alias locate locate -d /root/locate/db/locate.database.root

With the "-d /root/locate/db/locate.database.root" switch, locate will use the db at /root/locate/db/locate.database.root instead of the default /var/db/locate.database and root will be able to use locate to find any files in the filesystem, not just those that are world readable.

Finally, one way to update the regular locate db as root but without making it list every world readable file is to perform the following:

CODE:

#!/bin/sh
# make sure db file exists:
touch /var/db/locate.database

# then change ownership to the nobody user:
chown nobody /var/db/locate.database

# make it writeable by nobody and readable by everyone else:
chmod 644 /var/db/locate.database

# then move on to update the db...
# first make sure we're in the / folder where the db update starts:
cd /

# then finally run the updatedb command as the 'nobody' user:
echo "/usr/libexec/locate.updatedb" | su - -fm nobody

This is basically what the 310.locate periodic script does and results in a locate db that contains only files that are readable by the 'nobody' user - essentially all 'world readable' files.

Comparing the sizes of the root db against the nobody db:

CODE:

root@users /# ls -al /var/db/locate.database /root/locate/db/locate.database.root
-rw-r--r-- 1 root wheel 4070484 Nov 18 02:45 /root/locate/db/locate.database.root
-rw-r--r-- 1 nobody wheel 3280409 Nov 18 11:41 /var/db/locate.database

You can see the size difference there, not as many entries in nobody's db as root's. Just to double check:

CODE:

root@users /root# locate .cshrc.root
/root/bin/ktrace.out
/root/ktrace.out
/usr/local/etc/snort/ktrace.out
root@users /root# echo "locate ktrace.out" | su - -fm nobody
/usr/local/etc/snort/ktrace.out

So from that you can see that 'nobody' can see the ktrace.out files located in /root - apart from root of course :) Sorted.

FreeBSD 6.2 To Include Security Event Auditing

nospam@example.com (munk) — Tue, 14 Nov 2006 16:38:39 +0000

Just read an interesting article about the addition of 'Security Event Auditing' in FreeBSD 6.2. Until now FreeBSD hasn't had any really useful security auditing other than using 'accounting' to log all syscalls which at best was confusing when it came to working out who did what when and how.

At one time I installed a kernel module lrexec to log all system exec calls, but this was also quite a handful to configure scripts so they reported only on certain users. Hopefully this new security auditing daemon will make security auditing a lot easier on FreeBSD.

Read the article for more info on what's new:
Security Event Auditing in FreeBSD 6.2

Also of interest is the new addition to the FreeBSD handbook on security auditing:
FreeBSD Handbook: Security Event Auditing

Expand shell globs using 'ctrl-z'

nospam@example.com (munk) — Sun, 05 Nov 2006 11:22:58 +0000

Just noticed a semi useful feature of the CSH shell (shells in general? not tested it) whilst running 'rm -rf *' in a directory. Got a bit paranoid I was doing something silly (running the command as root), so hit 'ctrl-z' to put the process into the background and the '*' part was expanded in the job control list:

CODE:

[11:21:24] root@users /usr/local/www/web/torrentflux.munk.me.uk# rm -rf *
^Z
[1] + Suspended rm -rf TF_BitTornado adodb downloads images language mods searchEngines themes

Yay. Useful tip #341 you'll probably never use but at some point in the future think mmm... now where did I read about that thing about this thing...

Firefox 2.0 Released

nospam@example.com (munk) — Sat, 28 Oct 2006 01:57:00 +0000

UPDATE:
Here's a list of the pros I found about 2.0 so far:

Memory usage actually seems to have gone down somewhat - ok ok it probably couldn't have gotten any worse! I was half expecting memory to go through the roof with 2.0, but surprisingly it seems to actually free up memory ok when you close a lot of tabs or excess windows.
Generally speed wise, pages seem to load a lot faster than before.
Recently closed tabs are available via the history file menu. Can't remember that being there in 1.5.
The number of days of browsing history to keep track of also seems to be configurable in Firefox 2.0 - maybe it was in 1.5, can't remember seeing it obviously though.
Built in anti phishing tool is pretty nifty. Tried browsing to a phishing spam mail sent to my gmail account and Firefox spotted the site was fake, blacked the window out and popped up a warning instead with a few options on it - all in a jedi mind concentrate stylee. Smart.
Updates to extensions are now listed in a separate tab when they're found in the 'add ons' window (Tools, Options). Makes it easier than paging down through pages of extensions looking for the ones that have updates.
Interface is v snazmondo :D

And here's a list of the cons I have so far - so far it's mainly to do with tabs:

Tabs in 2.0 now have individual 'X'/close buttons on them. I find this quite annoying - apart from taking up extra space on the tab bar, the close buttons are now harder to hit with the mouse than they used to be. In 1.5 the X button was always to the far right of the tab bar, but now it's in a more random place on the tab bar depending on what tab you're reading/wanting to close. In 1.5 I'd just hover over to the right of the tab bar, whack on the close button and job done. Now though in 2.0 there's a split second where I have to think 'mmm where's the X this time then...?'. Minor annoyance but no doubt I'll get used to it.

UPDATE:As soon as I'd written this, sods law, the first thing I read was an article about pros/cons of Firefox 2.0 on Wired with a fix for the problem - just change 'browser.tabs.CloseButtons' in about:config to 3 instead of the default of 1. Still think this sucks though having to use about:config to change this kind of thing. Also there's something in those comments about setting Firefox to not resize images automatically - the option to do that has now dissappeared in Firefox 2.0 - you have to change stuff in about:config to fix the problem. That's another gripe I did notice but didn't think too much about last night.
Come to think about it I don't like the drop down button that lists all the current tabs open either. Again there's a fix but it involves editing the userChrome.css file found in the Firefox profile folder.

Here is a list of other tab tweaks for Firefox 2.0.
There's a new feature to allow Firefox to restart itself after extensions etc have installed themselves. Unfortunately though this useful feature doesn't seem to have a user interface - it's only available to extension installations (and themes?). It'd be good to be able to restart the browser when you make changes in the 'Addons' window - for example when you disable/enable some extension or another and want to restart the browser but keep your session going.
Too much faffing around in about:config!!!
Ok, yet another problem that needs config changing to fix it. I'm not mad about the 'restore from crash' functionality - if firefox crashes it automatically restores the last 'session' - group of windows/tabs. Unfortunately I often close my machine down using the power button on the PC and any open windows are 'gracelessly' killed - so next time I boot up, Firefox cheerfully asks me if I want to restore the crashed session which isn't really what I want and is quite annoying tbh.

The resolution is - surprise surprise yet another hack - to edit the config so the value of the boolean config option browser.sessionstore.resume_from_crash is set to false. In my case the key didn't even exist so I had to create it first and set it to 'false'. Haven't actually tested it out yet but presumably it works and doesn't give the annoying nag about resuming the session after a 'crash'.

Have to say all these config changes I've had to make so far are pretty tedious and if it weren't for this article I'd probably forget how I did it all if I needed to reinstall. By the same token though the very fact you can hack away at the config at all via about:config so easily is quite nice.

Mozilla have just released a new point version of Firefox, version 2.0. I held off installing and testing the beta/release candidate versions for a while because so many of the extensions I use wouldn't work with 2.0, but now 2.0 is officially out it seems most of my extensions now work ok.

There were 6 extensions that weren't compatible with 2.0 - most of them I could live without, but there was one, Clone Window, that I really couldn't do without. Clone window allows you to create a new tab/window with the history of the tab/window it was cloned from, without this the world just isn't right, bit like going out on the tiles for a few beers and waking up the next morning wondering how the fsck you ended up in a tent. :D

I ended up finding a similar extension called Duplicate Tabs which effectively does the same thing as Clone Window and more even. As well as creating clone tabs/windows, it even allows you to merge all open windows or a selection of open tabs into one window which is quite cool (not sure how much I'll use it but it's a good idea none the less).

Only trouble with the Duplicate Tabs extension is that the default key mapping to clone a tab - ctrl-shift-t - wouldn't work. Turns out that mapping was already in use by the Web Developer extension - tried changing the mapping in the web developer options, but that didn't seem to make any difference which was annoying.

Eventually tried to change the mapping using yet another extension called keyconfig - although annoyingly keyconfig wouldn't install because it wasn't compatible with 2.0! Grr... so I've now kind of hacked things up so it WOULD install by adding a boolean key extensions.checkCompatibility in about:config in Firefox and setting it to false - this stops Firefox checking the compatibility of extensions when they're installed or started with Firefox. Not entirely sure this is a good thing yet since it's now enabled all the extensions (those 6 above) that weren't compatible with 2.0 and there's a chance that might cause problems. Will see.

Generally though moving to Firefox 2.0 actually seems like a positive experience - browsing does seem to be faster in some unquantifiable way. Whether this is just the smoother looking interface making things seem quicker... I don't think it is... pages actually do seem to load faster now so I'm a happy bunny right now. Once all my extensions are compatible without forcing them I'll be even happier :)

If you've not upgraded as yet, I'd recommend it now it's offical.

I'll add any extra comments about my experience as I go on.

Modsecurity 2.0 Released

nospam@example.com (munk) — Fri, 20 Oct 2006 14:24:52 +0000

A new version of mod_security has just been released - 2.0 - complete with a total rewrite that includes a number of new features. El reg is running an article on the new release which includes an interview with ModSecurity's author Ivan Ristic.

mod_security is an apache module for monitoring requests made to a web server and acting on those requests according to rules - useful for blocking malicious bots, stopping web spammers and so on. I've been using it for a few years now and it handles blocking of weblog spammers and trojan worms/bots very well, though it has to be said the configuration isn't the simplest of all time.

Hopefully this configuration issue might be made easier with the also newly released modsecurity console, although reading through that page it doesn't seem to mention anything about using it to configure mod_security... Will have a look at it later and see what's what.

A list of the new features or improved features in ModSecurity 2.0 - taken from the article above:

Five processing phases (where there were only two in 1.9.x). These are: request headers, request body, response headers, response body, and logging. Those users who wanted to do things at the earliest possible moment can do them now.

Per-rule transformation options (previously normalisation was implicit and hard-coded). Many new transformation functions were added.

Transaction variables. This can be used to store pieces of data, create a transaction anomaly score, and so on.

Data persistence (can be configured any way you want although most people will want to use this feature to track IP addresses, application sessions, and application users).

Support for anomaly scoring and basic event correlation (counters can be automatically decreased over time; variables can be expired).

Support for web applications and session IDs.

Regular Expression back-references (allows one to create custom variables using transaction content).

There are now many functions that can be applied to the variables (where previously one could only use regular expressions).

XML support (parsing, validation, XPath).

The article is well worth reading if you already use ModSecurity - particularly if you're interested in moving from just simple blocking and logging of requests as in mod_security 1.0 to a more sophisticated web application firewalling system - mod_security 2.0. 2.0 includes a pseudo web app firewalling programming language making it easy to manipulate and process HTTP in a stateful manner - tracking HTTP sessions per IP in real time for example or perhaps watching for anomalous web activity and then flagging any IP that transgresses behaviour deemed as acceptable and watching for that IP in the future.

CSH Tips: Auto 'whereis' on the tcsh command line

nospam@example.com (munk) — Wed, 18 Oct 2006 00:10:26 +0000

Just read this entry about the 'whereis' command on Unix and it reminded me of another great shell tip for tcsh users (csh/tcsh on FreeBSD since they're the same thing!) - the shell can 'normalize' any command on the command line if you bind the normalize function to a keystroke - this allows you to easily see how a command would expand after it's executed. Somewhat esoteric without an example, but I use it so much I thought I'd post about it.

First off set a key binding for the normalize command - in the shell type in:

CODE:

bindkey "^W" normalize-command

or add it to ~/.cshrc to make it permanent. Obviously you can set it to whatever keybinding you want, I use ctrl-w.

Now type in any command that you'd use on the commandline and then whilst the cursor is at the end of the command, hit the keystroke you entered for the normalize command - in our case above, ctrl-w.

The command you entered should automatically get expanded to the absolute path of the command. For example if I type in:

CODE:

and then hit 'ctrl-w' whilst the cursor is just after the 's', the result will look like:

CODE:

/bin/ls

Magic!

Like I say I use this function quite a lot on the commandline, particularly when I want to see how a command I enter will get expanded - the normalize function works on aliases as well as just plain commands, so it's quicker to type 'portupgrade^w' than to type in 'alias portupgrade' to see how I've got my portupgrade alias set up. It's also great for quickly editing system executables from the commandline without having to remember where the file/script is or cut/paste the results from 'whereis' or 'locate' etc.

Shell Tip: Review your most commonly used command lines

nospam@example.com (munk) — Tue, 10 Oct 2006 00:22:47 +0000

Just got through reading an interesting article on how to review your most commonly used Unix commands. The idea is to sort the most commonly used commands numerically with a view to maybe shortening the most command ones using aliases, similar in a way to the time saving article mentioned here a while ago.

(The lifehacker article is actually just picking up on the original article on IBM's site entitled Unix productivity tips and is a good read for anyone wanting to improve their efficiency on the shell command line.)

Note for tcsh on FreeBSD, the command you probably want to use is this:

CODE:

history | tail -1000 | awk '{print $3}' | sort | uniq -c | sort -r

which outputs this kind of list for my shell history (listing top 10 used commands using '| head -10'):

CODE:

; history | tail -1000 | awk '{print $3}' | sort | uniq -c | sort -r | head -10
212 sc
158 m
  96 fg
  75 s
  68 cd
  56 vi
  37 ls
  26 grep
  19 man
  18 alias

I'm pretty happy with that, most of the commands are either 1 or 2 character aliases at least :)