Google the Site
Quicksearch
Categories
Syndicate This Blog
Creative Commons
Powered by
Blog Administration
Wednesday, August 15. 2007
Command Line Web Browsing With WWW::Mechanize::Shell
Introduction
The perl module WWW::Mechanize::Shell is a brilliant tool for browsing websites at a very low level - think somewhere in between using telnet and using a command line based browser like lynx or links or w3m and you'll be close. WWW::Mechanize::Shell is more than that though, it allows you to script a complete HTTP session so it can be replayed back at a later date without any interaction using WWW::Mechanize::Shell's parent perl module WWW::Mechanize - great for automatically submitting HTML forms/ POST data regularly via a cron job for example.
In this article I'll be talking about installing WWW::Mechanize::Shell, look at a typical WWW::Mechanize::Shell browsing session and look at some examples of how I use WWW::Mechanize::Shell to make things easier. Finally the article will end with a real world example - using mechshell to automate logging into FreshPorts and updating a watch list.
Installing WWW::Mechanize::Shell
As the name suggests, WWW::Mechanize::Shell is a perl module whose 'parent' is the WWW::Mechanize module written by Andy Lester (WWW::Mechanize::Shell itself is written by Max Maischein at time of original writing). WWW::Mechanize does all the work in the background - WWW::Mechanize::Shell just makes it easy to interact in a HTTP session. WWW::Mechanize::Shell and all it's dependencies can be installed from the ports tree:
Getting Started Using WWW::Mechanize::Shell
Once installed, start up the WWW::Mechanize::Shell using the following you can use the following commandline:
To make things easier though I use a CSH shell alias which aliases 'mechshell' to the command above:
Examples of WWW::Mechanize Usage
I usually use WWW::Mechanize when I want to manipulate data from websites that require a stateful HTTP session - ie a browsing session where there's more than one URL you have to visit to complete the 'session'. Usually these kind of stateful sessions involve logging into the website first, then browsing to another page to obtain the data and then I have the WWW::Mechanize perl script handle the data and return any results on the commandline.
Some examples of scripts that I've use WWW::Mechanize with:
eclipse_flex_speed.pl
My ISP (Eclipse UK) used to allow you to 'flex' your internet speed from 256k up to 2Mbps. They ran an offer for a while where you could flex to the max for 3 months - unfortunately you could only flex for 12 hours at a time, which meant logging into the control panel every 12hrs, selecting the maximum speed and then submitting the form. PITA basically.
Instead I wrote eclipse_flex_speed.pl to automatically login to the Eclipse control panel, 'click' the 2Mbps radio button and then submit the form so my speed got flexed automagically. I then added the script as a cron job to autorun every 12hours, saving the haslle of doing it all manually!
aod_get.pl
The BBC website allows you to listen to streams of all BBC radio broadcasts for up to a week after they've been aired live. The problem is that the web interface you listen to the stream on in your web browser only allows you to skip 5 or 15 minutes ahead in time and doesn't allow you to go to specific times in the stream. To get around this you can obtain the URL of the real player stream and open it in a standalone real player - doing this you can go to any point in the stream easily. Trouble is finding the URL of the stream isn't that easy and involves viewing the source HTML of the web UI and copy/pasting a partial URL.
I started to write a WWW::Mechanize script to automate the 'screen scraping' of all the available feeds from the BBC Audio On Demand site and listing them on one single HTML page linking the name of the feed to the real player feed URL. As it goes though, someone else - Dave Cross - already had the same idea and wrote a great script for scraping the BBC feeds automatically. I now run this in a cronjob once a week.
torrentflux_ctl.pl
This is a script for starting and stopping all torrents under the control of the torrentflux web based bittorrent client. The script logs in as the torrent owner and then stops or starts all the torrents for that user - basically just does a GET of a URL that causes torrentflux to stop or start all torrents. Crude but effective.
Real world example - Automating the update of FreshPorts watch list
Below is a real world example usage of WWW::Mechanize::Shell - automating the procedure of updating your watch list on Freshports.org. I've included comments as '# this is a comment' to help explain what each command is doing.
Now all that remains is to open up /tmp/freshports_update.pl and tidy the script up so that it's more suitable for automated use via cron. For example, any 'dump' and 'content' commands can be taken out - these would only cause problems anyway if run from a non-interactive shell as used by cron.
We also need to add some code to have the script dump the contents of 'pkg_info -qoa' to a temporary file prior to uploading.
The completed 'quick and dirty' hack looks like this then:
After saving the script and making the file executable, an entry can then be added to cron to have the script auto update the list of ports at freshports once a week - or however often you require it to be updated, once a week is more than enough for me. Sorted! :)
The perl module WWW::Mechanize::Shell is a brilliant tool for browsing websites at a very low level - think somewhere in between using telnet and using a command line based browser like lynx or links or w3m and you'll be close. WWW::Mechanize::Shell is more than that though, it allows you to script a complete HTTP session so it can be replayed back at a later date without any interaction using WWW::Mechanize::Shell's parent perl module WWW::Mechanize - great for automatically submitting HTML forms/ POST data regularly via a cron job for example.
In this article I'll be talking about installing WWW::Mechanize::Shell, look at a typical WWW::Mechanize::Shell browsing session and look at some examples of how I use WWW::Mechanize::Shell to make things easier. Finally the article will end with a real world example - using mechshell to automate logging into FreshPorts and updating a watch list.
Installing WWW::Mechanize::Shell
As the name suggests, WWW::Mechanize::Shell is a perl module whose 'parent' is the WWW::Mechanize module written by Andy Lester (WWW::Mechanize::Shell itself is written by Max Maischein at time of original writing). WWW::Mechanize does all the work in the background - WWW::Mechanize::Shell just makes it easy to interact in a HTTP session. WWW::Mechanize::Shell and all it's dependencies can be installed from the ports tree:
CODE:
root@users /root# cd /usr/ports/www/p5-WWW-Mechanize-Shell/
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# make install
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# rehash
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# make install
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# rehash
Getting Started Using WWW::Mechanize::Shell
Once installed, start up the WWW::Mechanize::Shell using the following you can use the following commandline:
CODE:
root@users /root# perl -MWWW::Mechanize::Shell -eshell
To make things easier though I use a CSH shell alias which aliases 'mechshell' to the command above:
CODE:
root@users /root# grep mechshell $cshrc
alias mechshell perl -MWWW::Mechanize::Shell -eshell
alias mechshell perl -MWWW::Mechanize::Shell -eshell
Examples of WWW::Mechanize Usage
I usually use WWW::Mechanize when I want to manipulate data from websites that require a stateful HTTP session - ie a browsing session where there's more than one URL you have to visit to complete the 'session'. Usually these kind of stateful sessions involve logging into the website first, then browsing to another page to obtain the data and then I have the WWW::Mechanize perl script handle the data and return any results on the commandline.
Some examples of scripts that I've use WWW::Mechanize with:
eclipse_flex_speed.pl
My ISP (Eclipse UK) used to allow you to 'flex' your internet speed from 256k up to 2Mbps. They ran an offer for a while where you could flex to the max for 3 months - unfortunately you could only flex for 12 hours at a time, which meant logging into the control panel every 12hrs, selecting the maximum speed and then submitting the form. PITA basically.
Instead I wrote eclipse_flex_speed.pl to automatically login to the Eclipse control panel, 'click' the 2Mbps radio button and then submit the form so my speed got flexed automagically. I then added the script as a cron job to autorun every 12hours, saving the haslle of doing it all manually!
aod_get.pl
The BBC website allows you to listen to streams of all BBC radio broadcasts for up to a week after they've been aired live. The problem is that the web interface you listen to the stream on in your web browser only allows you to skip 5 or 15 minutes ahead in time and doesn't allow you to go to specific times in the stream. To get around this you can obtain the URL of the real player stream and open it in a standalone real player - doing this you can go to any point in the stream easily. Trouble is finding the URL of the stream isn't that easy and involves viewing the source HTML of the web UI and copy/pasting a partial URL.
I started to write a WWW::Mechanize script to automate the 'screen scraping' of all the available feeds from the BBC Audio On Demand site and listing them on one single HTML page linking the name of the feed to the real player feed URL. As it goes though, someone else - Dave Cross - already had the same idea and wrote a great script for scraping the BBC feeds automatically. I now run this in a cronjob once a week.
torrentflux_ctl.pl
This is a script for starting and stopping all torrents under the control of the torrentflux web based bittorrent client. The script logs in as the torrent owner and then stops or starts all the torrents for that user - basically just does a GET of a URL that causes torrentflux to stop or start all torrents. Crude but effective.
Real world example - Automating the update of FreshPorts watch list
Below is a real world example usage of WWW::Mechanize::Shell - automating the procedure of updating your watch list on Freshports.org. I've included comments as '# this is a comment' to help explain what each command is doing.
CODE:
# Start up mechshell - alias for 'perl -MWWW::Mechanize::Shell -eshell':
munk@users /home/munk# mechshell
# Request the URL http://www.freshports.org/login.php.
# Note the HTTP response '(200)' is displayed underneath
# to indicate the page was fetched successfully:
(no url)>get http://www.freshports.org/login.php
Retrieving http://www.freshports.org/login.php(200)
# Use the mechshell 'dump' command to dump the contents
# of all forms found on the login page:
http://www.freshports.org/login.php>dump
POST http://www.freshports.org/login.php?origin=%2F [l]
custom_settings=1 (hidden readonly)
LOGIN=1 (hidden readonly)
UserID= (text)
Password= (password)
submit=Login (submit)
<NONAME>=<UNDEF> (reset)
# There's just a single form on this page:
# - the form's 'ACTION' is set to submit the form using the POST method
# to the url http://www.freshports.org/login.php?origin=%2F
# The form contains the following form fields:
# - 2 hidden fields
# - 1 text field called 'UserID'
# - 1 password field called 'Password'
# - 1 submit field called 'Login'
# Fill in the 'UserID' and 'Password' fields:
http://www.freshports.org/login.php>value UserID munk
http://www.freshports.org/login.php>value Password xxxxxx
# And then submit the form. Note we can just use the mechshell 'submit'
# command here because there is only a single form on the page. If there were
# more than one form on the page we would need to specify which button exactly to
# click:
http://www.freshports.org/login.php>submit
200
# Again note that the '200' response indicates the request was successful.
# Also note that the next mechshell prompt below has changed from
# 'http://www.freshports.org/login.php>' to just 'http://www.freshports.org/' -
# this indicates that the login script has probably redirected us to the
# freshports home page.
# Now we take a look to check that the login succeeded ok. To do this we use
# the mechshell 'content' command which effectively dumps the content of the
# returned page back at us in a pager.
# What we're looking for is the text 'Logged in as munk' which will indicate we
# logged in ok:
http://www.freshports.org/>content
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
-snip-
<td NOWRAP><FONT SIZE="-1">Logged in as munk</FONT><br><FONT SIZE="-1"><a href="/customize.php?origin=%2F" title="Customize your settings">Customize</a
></FONT><br><FONT SIZE="-1"><a href="/logout.php" title="Logout of the website">Logout</a></FONT><br><FONT SIZE="-1"><a href="/my-flagged-commits.php" title="Li
st of commits you have flagged">My Flagged Commits</a></FONT><br>
-snip-
# Now we're logged in ok we can continue to upload the mypkg_info.txt file we
# created earlier.
# First browse to the pkg_upload.php page:
http://www.freshports.org/>get http://www.freshports.org/pkg_upload.php
Retrieving http://www.freshports.org/pkg_upload.php(200)
# Now use 'dump' to see a list of form fields on this page.
# Note that there are 2 submit buttons on this page:
http://www.freshports.org/pkg_upload.php>dump
POST http://www.freshports.org/pkg_upload.php (multipart/form-data)
pkg_info= (file)
staging=Staging (submit)
wlid=5393 (option) [*5393/main*]
replaceappend=replace (radio) [*replace/Replace list contents|append/Append to list (duplicates will be removed)]
upload=Upload (submit)
# We need to fill out the form here. Uploading files with mechshell is as
# simple as completing the correct file type field:
http://www.freshports.org/pkg_upload.php>value pkg_info /tmp/mypkg_info.txt
# Ok, now we're ready to submit the form.
# Note that because there are 2 submit buttons on this form, we must explicitly
# tell mechshell which button it is that we want to click on - to do that we use
# the 'click' command. Just using 'submit' here would possibly click on the
# 'staging' button which is not what we want - instead we use the command
# 'click upload' to indicate we want to click on the 'upload' button:
http://www.freshports.org/pkg_upload.php>click upload
(200)
# Success! It's a good idea now to just check that this worked by browsing in
# a web browser to your watch list and checking the new items were updated ok (of
# course you can do this in mechshell if you want but I'll leave that out here!).
# Finally, the really cool bit. The mechshell 'script' command will dump out
# the perl code required to perform all of the above actions again if you copy
# them into a perl script:
http://www.freshports.org/pkg_upload.php>script
#!perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
-snip-
# Also, if you provide a filename as an argument to the 'script' command,
# mechshell will dump all the script commands to that filename:
http://www.freshports.org/pkg_upload.php>script /tmp/freshports_update.pl
# Finally, use 'quit' to exit the mechshell:
http://www.freshports.org/pkg_upload.php>quit
munk@users /home/munk#
munk@users /home/munk# mechshell
# Request the URL http://www.freshports.org/login.php.
# Note the HTTP response '(200)' is displayed underneath
# to indicate the page was fetched successfully:
(no url)>get http://www.freshports.org/login.php
Retrieving http://www.freshports.org/login.php(200)
# Use the mechshell 'dump' command to dump the contents
# of all forms found on the login page:
http://www.freshports.org/login.php>dump
POST http://www.freshports.org/login.php?origin=%2F [l]
custom_settings=1 (hidden readonly)
LOGIN=1 (hidden readonly)
UserID= (text)
Password= (password)
submit=Login (submit)
<NONAME>=<UNDEF> (reset)
# There's just a single form on this page:
# - the form's 'ACTION' is set to submit the form using the POST method
# to the url http://www.freshports.org/login.php?origin=%2F
# The form contains the following form fields:
# - 2 hidden fields
# - 1 text field called 'UserID'
# - 1 password field called 'Password'
# - 1 submit field called 'Login'
# Fill in the 'UserID' and 'Password' fields:
http://www.freshports.org/login.php>value UserID munk
http://www.freshports.org/login.php>value Password xxxxxx
# And then submit the form. Note we can just use the mechshell 'submit'
# command here because there is only a single form on the page. If there were
# more than one form on the page we would need to specify which button exactly to
# click:
http://www.freshports.org/login.php>submit
200
# Again note that the '200' response indicates the request was successful.
# Also note that the next mechshell prompt below has changed from
# 'http://www.freshports.org/login.php>' to just 'http://www.freshports.org/' -
# this indicates that the login script has probably redirected us to the
# freshports home page.
# Now we take a look to check that the login succeeded ok. To do this we use
# the mechshell 'content' command which effectively dumps the content of the
# returned page back at us in a pager.
# What we're looking for is the text 'Logged in as munk' which will indicate we
# logged in ok:
http://www.freshports.org/>content
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
-snip-
<td NOWRAP><FONT SIZE="-1">Logged in as munk</FONT><br><FONT SIZE="-1"><a href="/customize.php?origin=%2F" title="Customize your settings">Customize</a
></FONT><br><FONT SIZE="-1"><a href="/logout.php" title="Logout of the website">Logout</a></FONT><br><FONT SIZE="-1"><a href="/my-flagged-commits.php" title="Li
st of commits you have flagged">My Flagged Commits</a></FONT><br>
-snip-
# Now we're logged in ok we can continue to upload the mypkg_info.txt file we
# created earlier.
# First browse to the pkg_upload.php page:
http://www.freshports.org/>get http://www.freshports.org/pkg_upload.php
Retrieving http://www.freshports.org/pkg_upload.php(200)
# Now use 'dump' to see a list of form fields on this page.
# Note that there are 2 submit buttons on this page:
http://www.freshports.org/pkg_upload.php>dump
POST http://www.freshports.org/pkg_upload.php (multipart/form-data)
pkg_info= (file)
staging=Staging (submit)
wlid=5393 (option) [*5393/main*]
replaceappend=replace (radio) [*replace/Replace list contents|append/Append to list (duplicates will be removed)]
upload=Upload (submit)
# We need to fill out the form here. Uploading files with mechshell is as
# simple as completing the correct file type field:
http://www.freshports.org/pkg_upload.php>value pkg_info /tmp/mypkg_info.txt
# Ok, now we're ready to submit the form.
# Note that because there are 2 submit buttons on this form, we must explicitly
# tell mechshell which button it is that we want to click on - to do that we use
# the 'click' command. Just using 'submit' here would possibly click on the
# 'staging' button which is not what we want - instead we use the command
# 'click upload' to indicate we want to click on the 'upload' button:
http://www.freshports.org/pkg_upload.php>click upload
(200)
# Success! It's a good idea now to just check that this worked by browsing in
# a web browser to your watch list and checking the new items were updated ok (of
# course you can do this in mechshell if you want but I'll leave that out here!).
# Finally, the really cool bit. The mechshell 'script' command will dump out
# the perl code required to perform all of the above actions again if you copy
# them into a perl script:
http://www.freshports.org/pkg_upload.php>script
#!perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
-snip-
# Also, if you provide a filename as an argument to the 'script' command,
# mechshell will dump all the script commands to that filename:
http://www.freshports.org/pkg_upload.php>script /tmp/freshports_update.pl
# Finally, use 'quit' to exit the mechshell:
http://www.freshports.org/pkg_upload.php>quit
munk@users /home/munk#
Now all that remains is to open up /tmp/freshports_update.pl and tidy the script up so that it's more suitable for automated use via cron. For example, any 'dump' and 'content' commands can be taken out - these would only cause problems anyway if run from a non-interactive shell as used by cron.
We also need to add some code to have the script dump the contents of 'pkg_info -qoa' to a temporary file prior to uploading.
The completed 'quick and dirty' hack looks like this then:
CODE:
#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
# FreshPorts username/pass:
my $user="munk";
my $pass="xxxxx";
# Temp location to store output from 'pkg_info -qoa':
my $mypkg_info="/tmp/freshports/mypkg_info.txt";
# prepare file containing output from: pkg_info -qoa
`pkg_info -qoa > $mypkg_info`;
# Prepare WWW::Mechanize:
my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
$agent->env_proxy();
# Login to FreshPorts:
$agent->get('http://www.freshports.org/login.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('UserID', $user); };
{ local $^W; $agent->current_form->value('Password', $pass); };
$agent->submit();
# Submit pkg_info details to FreshPorts pkg_upload page:
$agent->get('http://www.freshports.org/pkg_upload.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('pkg_info', $mypkg_info); };
$agent->click('upload');
# Remove temporary file:
`rm $mypkg_info`;
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
# FreshPorts username/pass:
my $user="munk";
my $pass="xxxxx";
# Temp location to store output from 'pkg_info -qoa':
my $mypkg_info="/tmp/freshports/mypkg_info.txt";
# prepare file containing output from: pkg_info -qoa
`pkg_info -qoa > $mypkg_info`;
# Prepare WWW::Mechanize:
my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
$agent->env_proxy();
# Login to FreshPorts:
$agent->get('http://www.freshports.org/login.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('UserID', $user); };
{ local $^W; $agent->current_form->value('Password', $pass); };
$agent->submit();
# Submit pkg_info details to FreshPorts pkg_upload page:
$agent->get('http://www.freshports.org/pkg_upload.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('pkg_info', $mypkg_info); };
$agent->click('upload');
# Remove temporary file:
`rm $mypkg_info`;
After saving the script and making the file executable, an entry can then be added to cron to have the script auto update the list of ports at freshports once a week - or however often you require it to be updated, once a week is more than enough for me. Sorted! :)
Wednesday, January 17. 2007
Installing Exim, SASLAuthd, ClamAV and SpamAssassin on FreeBSD 6.2
Introduction
This article describes the steps necessary to install and configure Exim on FreeBSD 6.2 with support for the following:
Each of the required 'dependencies' or components will be installed and configured, Exim will be installed and configured and finally we will test to check each component is working as required.
With regards to spam and malware scanning, the system described will quarantine any files/messages that it finds classified as spam or malware. In this way the quarantined files can be checked over by the admin at a later date and various stats gathering can be done if required.
Installing and Configuring SASLAuthd
SASLAuthd is an authentication daemon that can handle authentication requests from 3rd party applications such as Exim - generally for any application that can't directly access a system password database because of permission restrictions. In this case running Exim MTA as 'root' is a potential security risk, so exim runs as the 'mailnull' user on FreeBSD. Unfortunately this means Exim can't easily read the system password database to authenticate users who want to send mail via the server, which is where SASLAuthd comes in. Any requests for authentication with Exim are passed on to the SASLAuthd daemon which will then verify whether the user credentials are valid - if so, the email is delivered, if not, it's rejected.
Installing and Configuring SpamAssassin
SpamAssassin (SA) is one solution to the problem of spam. SA can run as a daemon (spamd) in the background and accept requests from an MTA such as Exim to check whether an email message should be classified as spam.
Spamd looks at the message and checks for various factors that make the message more or less likely to be spam and assigns the message a score based on what it finds. Spamd will then reply to the MTA, telling it the spam score that it gave that message. The MTA can then decide - based on that score - whether to accept/reject the message - or in the case of this guide whether to instead quarantine the message.
Installing and Configuring ClamAV
ClamAV is an anti-virus suite and includes a daemon clamd (runs in the background to check for requests to test for virii), another daemon freshclam (updates the virus definition database) and a couple of clients to run on the commandline if you need them for local virus scanning.
Exim will send requests to the clamd server in much the same was as spamd does - if clamd classifies a message as containing a virus, Exim will reject delivery of the message and instaed quarantine it.
ClamAV should be configured now and ready to accept request to check for malware from the Exim MTA.
We can now move on finally to install and configure Exim.
Installing and Configuring Exim
Exim configuration can be very complicated. This guide will only deal with the configuration of Exim so it accepts mail on a domain 'mail.example.com', scans the mail for malware/spam - quarantining anything it finds as malware/spam and accepts authentication requests correctly.
Important:
Ensure your mail server's DNS is configured correctly and preferably has a reverse DNS record (rDNS) set up. Many mail servers will not deliver mail correctly to/from your mail server without rDNS.
Configuring Exim
We now move on to configuring Exim.
Exim should now be set to check for malware/spam and to authenticate users.
Testing Exim configuration
Finally we can move on to test that our config works correctly for spam/malware checking and for authenticating users.
So we now have a working Exim with support for spam/malware checking and authentication over SMTP.
This article describes the steps necessary to install and configure Exim on FreeBSD 6.2 with support for the following:
- authenticated SMTP (asmtp) using SASLAuthd
spam detection and quarantine using SpamAssassin
malware detection and quarantine using ClamAV
Each of the required 'dependencies' or components will be installed and configured, Exim will be installed and configured and finally we will test to check each component is working as required.
With regards to spam and malware scanning, the system described will quarantine any files/messages that it finds classified as spam or malware. In this way the quarantined files can be checked over by the admin at a later date and various stats gathering can be done if required.
Installing and Configuring SASLAuthd
SASLAuthd is an authentication daemon that can handle authentication requests from 3rd party applications such as Exim - generally for any application that can't directly access a system password database because of permission restrictions. In this case running Exim MTA as 'root' is a potential security risk, so exim runs as the 'mailnull' user on FreeBSD. Unfortunately this means Exim can't easily read the system password database to authenticate users who want to send mail via the server, which is where SASLAuthd comes in. Any requests for authentication with Exim are passed on to the SASLAuthd daemon which will then verify whether the user credentials are valid - if so, the email is delivered, if not, it's rejected.
- Install SASLAuthd from the FreeBSD ports tree:
CODE:
root@win /root# cd /usr/ports/security/cyrus-sasl2-saslauthd/
root@win /usr/ports/security/cyrus-sasl2-saslauthd# make install
...
root@win /usr/ports/security/cyrus-sasl2-saslauthd# rehash
root@win /usr/ports/security/cyrus-sasl2-saslauthd# make install
...
root@win /usr/ports/security/cyrus-sasl2-saslauthd# rehash
Configure SASLAuthd to run at boot.
Edit /etc/rc.conf to include the following:
CODE:
saslauthd_enable="YES"
saslauthd_flags="-a getpwent"
saslauthd_flags="-a getpwent"
Note:
SASLAuthd will run using the 'getpwent' authentication mechanism with the flag above. This method uses the passwd file directly instead of using other means like kerberos or PAM. If you require another method, check the manpage for saslauthd.
Start the SASLAuthd daemon running:
CODE:
root@win /usr/ports/security/cyrus-sasl2-saslauthd# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# ./saslauthd start
root@win /usr/local/etc/rc.d# ./saslauthd start
(Optional) Test the SASLAuthd daemon:
Substitute 'user' and 'pass' for the username and password of a user
account on your system:
CODE:
root@win /usr/local/etc/rc.d# testsaslauthd -u user -p pass
0: OK "Success."
0: OK "Success."
Installing and Configuring SpamAssassin
SpamAssassin (SA) is one solution to the problem of spam. SA can run as a daemon (spamd) in the background and accept requests from an MTA such as Exim to check whether an email message should be classified as spam.
Spamd looks at the message and checks for various factors that make the message more or less likely to be spam and assigns the message a score based on what it finds. Spamd will then reply to the MTA, telling it the spam score that it gave that message. The MTA can then decide - based on that score - whether to accept/reject the message - or in the case of this guide whether to instead quarantine the message.
- Install SA from the FreeBSD ports.
Note:
There are various installation options you can choose when installing SA which you should see when you first run 'make install' in the SA port directory. To see the options after already configuring them you can run 'make config'.
In turn, each of SA's dependencies may also have options you can configure at install time.
To write this guide I'm only using the single option 'AS_ROOT' in the SA install configuation and for the other items generally just choose the
defaults.
CODE:
root@win /root# cd /usr/ports/mail/p5-Mail-SpamAssassin/
root@win /usr/ports/mail/p5-Mail-SpamAssassin# make install
root@win /usr/ports/mail/p5-Mail-SpamAssassin# make install
Once complete, you should see:
CODE:
*************************************************************************
* _ _____ _____ _____ _ _ _____ ___ ___ _ _ *
* / \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | | *
* / _ \ | | | | | _| | \| | | | | | | | | \| | *
* / ___ \| | | | | |___| |\ | | | | | |_| | |\ | *
* /_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_| *
* *
* See /usr/local/share/doc/p5-Mail-SpamAssassin/INSTALL, *
* and /usr/local/share/doc/p5-Mail-SpamAssassin/UPGRADE, *
* or http://spamassassin.org/dist/INSTALL and *
* http://spamassassin.org/dist/UPGRADE BEFORE enabling *
* this version of SpamAssassin for important information *
* regarding changes in this version. *
*************************************************************************
* _ _____ _____ _____ _ _ _____ ___ ___ _ _ *
* / \|_ _|_ _| ____| \ | |_ _|_ _/ _ \| \ | | *
* / _ \ | | | | | _| | \| | | | | | | | | \| | *
* / ___ \| | | | | |___| |\ | | | | | |_| | |\ | *
* /_/ \_\_| |_| |_____|_| \_| |_| |___\___/|_| \_| *
* *
* See /usr/local/share/doc/p5-Mail-SpamAssassin/INSTALL, *
* and /usr/local/share/doc/p5-Mail-SpamAssassin/UPGRADE, *
* or http://spamassassin.org/dist/INSTALL and *
* http://spamassassin.org/dist/UPGRADE BEFORE enabling *
* this version of SpamAssassin for important information *
* regarding changes in this version. *
*************************************************************************
It's a good idea to read the files listed in the banner above. SA has a large number of options that can be configured; a good place to start configuring options on FreeBSD is in /usr/local/etc/mail/spamassassin/.
Configure SA to run at boot.
Edit /etc/rc.conf to include the following:
CODE:
spamd_enable="YES"
Start SA spamd.
We can now go on to actually start spamd running as a daemon and verify spamd started ok:
CODE:
root@win /root# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# rehash
root@win /usr/local/etc/rc.d# ./sa
sa-spamd* saslauthd*
root@win /usr/local/etc/rc.d# ./sa-spamd start
Starting spamd.
munk@win /usr/local/etc/rc.d# ./sa-spamd status
spamd is running as pid 754.
root@win /usr/local/etc/rc.d# rehash
root@win /usr/local/etc/rc.d# ./sa
sa-spamd* saslauthd*
root@win /usr/local/etc/rc.d# ./sa-spamd start
Starting spamd.
munk@win /usr/local/etc/rc.d# ./sa-spamd status
spamd is running as pid 754.
This tells us spamd is running ok in the background.
Installing and Configuring ClamAV
ClamAV is an anti-virus suite and includes a daemon clamd (runs in the background to check for requests to test for virii), another daemon freshclam (updates the virus definition database) and a couple of clients to run on the commandline if you need them for local virus scanning.
Exim will send requests to the clamd server in much the same was as spamd does - if clamd classifies a message as containing a virus, Exim will reject delivery of the message and instaed quarantine it.
- Install ClamAV from the FreeBSD ports tree:
CODE:
root@win /root# cd /usr/ports/security/clamav
root@win /usr/ports/security/clamav# make install
root@win /usr/ports/security/clamav# make install
Configure ClamAV to start at boot time.
Edit /etc/rc.conf to include:
CODE:
clamav_clamd_enable="YES"
clamav_freshclam_enable="YES"
clamav_freshclam_enable="YES"
Configure clamd.
Edit /usr/local/etc/clamd.conf to include the following:
CODE:
LogFile /var/log/clamav/clamd.log
PidFile /var/run/clamav/clamd.pid
DatabaseDirectory /var/db/clamav
LocalSocket /var/run/clamav/clamd
FixStaleSocket
User clamav
AllowSupplementaryGroups
ScanMail
ScanArchive
PidFile /var/run/clamav/clamd.pid
DatabaseDirectory /var/db/clamav
LocalSocket /var/run/clamav/clamd
FixStaleSocket
User clamav
AllowSupplementaryGroups
ScanMail
ScanArchive
Start clamd and freshclam.
CODE:
root@win /root# cd /usr/local/etc/rc.d
root@win /usr/local/etc/rc.d# ./clamav-clamd start
Starting clamav_clamd.
root@win /usr/local/etc/rc.d# ./clamav-freshclam start
Starting clamav_freshclam.
root@win /usr/local/etc/rc.d# ./clamav-clamd start
Starting clamav_clamd.
root@win /usr/local/etc/rc.d# ./clamav-freshclam start
Starting clamav_freshclam.
Note:
You may see the following message on first running clamd:
CODE:
LibClamAV Warning: **************************************************
LibClamAV Warning: *** The virus database is older than 7 days. ***
LibClamAV Warning: *** Please update it IMMEDIATELY! ***
LibClamAV Warning: **************************************************
LibClamAV Warning: *** The virus database is older than 7 days. ***
LibClamAV Warning: *** Please update it IMMEDIATELY! ***
LibClamAV Warning: **************************************************
As long as you're running freshclam, you can safely ignore this message. Freshclam should update your definitions automatically. Be sure to configure freshclam to update the virus definitions regularly.
ClamAV should be configured now and ready to accept request to check for malware from the Exim MTA.
We can now move on finally to install and configure Exim.
Installing and Configuring Exim
Exim configuration can be very complicated. This guide will only deal with the configuration of Exim so it accepts mail on a domain 'mail.example.com', scans the mail for malware/spam - quarantining anything it finds as malware/spam and accepts authentication requests correctly.
Important:
Ensure your mail server's DNS is configured correctly and preferably has a reverse DNS record (rDNS) set up. Many mail servers will not deliver mail correctly to/from your mail server without rDNS.
- Install Exim from the FreeBSD ports tree:
CODE:
[12:10:57] root@win /root# cd /usr/ports/mail/exim
[12:12:30] root@win /usr/ports/mail/exim# make -DWITH_CONTENT_SCAN -DWITH_SASLAUTHD install
[12:12:30] root@win /usr/ports/mail/exim# make -DWITH_CONTENT_SCAN -DWITH_SASLAUTHD install
Stop the Sendmail daemon if it's already running:
CODE:
root@win /root# cd /etc/rc.d
root@win /etc/rc.d# ./sendmail stop
root@win /etc/rc.d# ./sendmail stop
Configure Exim to run at boot time.
Edit /etc/rc.conf to include:
CODE:
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
exim_enable="YES"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
exim_enable="YES"
This has the effect of disabling sendmail at boot time - the default FreeBSD MTA - and running Exim instead.
Configure mailer.conf to use Exim as the default MTA.
Edit /etc/mail/mailer.conf to read:
CODE:
sendmail /usr/local/sbin/exim
send-mail /usr/local/sbin/exim
mailq /usr/local/sbin/exim -bp
newaliases /usr/bin/true
send-mail /usr/local/sbin/exim
mailq /usr/local/sbin/exim -bp
newaliases /usr/bin/true
This will allow any FreeBSD base system mail related commands to use Exim instead of Sendmail.
Configuring Exim
We now move on to configuring Exim.
- Set the primary hostname.
Edit /usr/local/etc/exim/configure.
Find and edit the 'primary_hostname' line for your domain:
CODE:primary_hostname = example.com
This configures Exim to accept mail primarily for the 'example.com' domain - ie foobar@example.com.
Find and edit the following lines to read:
CODE:
av_scanner = clamd:/var/run/clamav/clamd
spamd_address = 127.0.0.1 783
spamd_address = 127.0.0.1 783
Configure the malware and spam Access Control Lists (ACLs).
How malware/spam checking works in this system:
We add a check in the acl_check_data ACL for spam and malware. Exim will request each email is checked for spam/malware by the relevant daemon - spamd for spam, clamd for malware. If the message is classified as spam/malware by the relevant daemons, Exim will add a header to the message 'X-Quarantine-Me-Spam' (similar for malware).
Later on when it comes to actually delivering (termed 'routing' in Exim terminology), we add two routers to test for the existence of the headers that are added in the acl_check_data ACL if a message is found to be spam/malware. If the headers are found by the malware/spam routers, the message is not delivered but instead copied to a quarantine location on disk.
This quarantine location can then be checked later by an admin to check if anything is amiss - ie regular non spam/malware mail that should really have been delivered.
Once you're satisfied the configuration is working as it should - ie after a few months of operation - and not finding false positives, you can change the malware/spam acl checks to just deny instead of adding the quarantine headers. Having said that, I still opt to just quarantine malware/spam and remove it at a later date.
On to configuring the data ACL:
Modify the acl_check_data ACL to read/include:
CODE:
acl_check_data:
# Deny if the message contains a virus. Before enabling this check, you
# must install a virus scanner and set the av_scanner option above.
#
# defer_ok - pass this message if scanner is down etc:
warn message = X-Quarantine-Me-Malware: $malware_name
log_message = malware: $malware_name
demime = *
malware = */defer_ok
# Add headers to a message if it is judged to be spam. Before enabling this,
# you must install SpamAssassin. You may also need to set the spamd_address
# option above.
#
warn message = X-Quarantine-Me-Spam: SA score $spam_score\n\
X-SA-Report: $spam_report
log_message = Spam score $spam_score > 5
spam = spamd/defer_ok
condition = ${if >{$spam_score_int}{50}{1}{0}}
# Accept the message.
accept
# Deny if the message contains a virus. Before enabling this check, you
# must install a virus scanner and set the av_scanner option above.
#
# defer_ok - pass this message if scanner is down etc:
warn message = X-Quarantine-Me-Malware: $malware_name
log_message = malware: $malware_name
demime = *
malware = */defer_ok
# Add headers to a message if it is judged to be spam. Before enabling this,
# you must install SpamAssassin. You may also need to set the spamd_address
# option above.
#
warn message = X-Quarantine-Me-Spam: SA score $spam_score\n\
X-SA-Report: $spam_report
log_message = Spam score $spam_score > 5
spam = spamd/defer_ok
condition = ${if >{$spam_score_int}{50}{1}{0}}
# Accept the message.
accept
At the top of the routers section, modify to read/include:
CODE:
begin routers
check_malware:
driver = redirect
condition = ${if def:h_X-Quarantine-Me-Malware: {1}{0}}
headers_add = X-Quarantined-Malware: $h_X-Quarantine-Me-Malware:
headers_remove = X-Quarantine-Me-Malware
data = /var/quarantine/malware/malware.$tod_logfile
file_transport = address_file
check_spam:
driver = redirect
condition = ${if def:h_X-Quarantine-Me-Spam: {1}{0}}
headers_add = X-Quarantined-Spam: $h_X-Quarantine-Me-Spam:
headers_remove = X-Quarantine-Me-Spam
data = /var/quarantine/spam/spam.$tod_logfile
file_transport = address_file
no_more
check_malware:
driver = redirect
condition = ${if def:h_X-Quarantine-Me-Malware: {1}{0}}
headers_add = X-Quarantined-Malware: $h_X-Quarantine-Me-Malware:
headers_remove = X-Quarantine-Me-Malware
data = /var/quarantine/malware/malware.$tod_logfile
file_transport = address_file
check_spam:
driver = redirect
condition = ${if def:h_X-Quarantine-Me-Spam: {1}{0}}
headers_add = X-Quarantined-Spam: $h_X-Quarantine-Me-Spam:
headers_remove = X-Quarantine-Me-Spam
data = /var/quarantine/spam/spam.$tod_logfile
file_transport = address_file
no_more
Modify the authenticators section to read:
CODE:
begin authenticators
plain:
driver = plaintext
public_name = PLAIN
server_condition = ${if saslauthd{{$2}{$3}}{1}{0}}
login:
driver = plaintext
public_name = LOGIN
server_prompts = "Username:: : Password::"
server_condition = ${if saslauthd{{$1}{$2}}{1}{0}}
plain:
driver = plaintext
public_name = PLAIN
server_condition = ${if saslauthd{{$2}{$3}}{1}{0}}
login:
driver = plaintext
public_name = LOGIN
server_prompts = "Username:: : Password::"
server_condition = ${if saslauthd{{$1}{$2}}{1}{0}}
Save the /usr/local/etc/exim/configuration file.
Create the quarantine directories and change ownership to mailnull:mail:
CODE:
root@win /root# mkdir -p /var/quarantine/{malware,spam}
root@win /root# chown mailnull:mail /var/quarantine/{malware,spam}
root@win /root# chown mailnull:mail /var/quarantine/{malware,spam}
Restart Exim to suck in the new config options:
CODE:
root@win /root# /usr/local/etc/rc.d/exim restart
Stopping exim.
Starting exim.
Stopping exim.
Starting exim.
Exim should now be set to check for malware/spam and to authenticate users.
Testing Exim configuration
Finally we can move on to test that our config works correctly for spam/malware checking and for authenticating users.
- Testing Exim's malware/spam scanning.
The easiest option is to send an email to your mailserver with specially crafted malware/spam signatures included in the body of the message. When spamd/clamd see these signature strings in the body of the messages, they should classify the message as spam/malware and Exim in turn will quarantine the messages.
The official EICAR malware/virus testing signature is as follows:
CODE:
X5O%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILEspamcH+H*
See here for the official string:
http://www.eicar.org/anti_virus_test_file.htm
The official GTUBE spam testing signature is as follows:
CODE:
XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X
See here for the official string:
http://spamassassin.apache.org/gtube/
Note:
Another option for testing spam/malware scanning is to run exim from the commandline using the command 'exim -bh 127.0.0.1'. This will run an SMTP session from the commandline (think telnet) and allow you to inject your own specially crafted message using the signatures above. This requires you enter a valid SMTP session, something like:
CODE:
HELO example.com
MAIL FROM:foo@example.com
RCPT TO:foo@example.com
DATA
X5O%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILEspamcH+H*
.
MAIL FROM:foo@example.com
RCPT TO:foo@example.com
DATA
X5O%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILEspamcH+H*
.
This would simulate the injection of a mail message with a virus in it and in 'exim -bh' mode you can see a lot of useful debugging info to verify everything works ok.
Testing Exim's Authentication configuration.
We can now test that ASMTP is working. For this you can either run exim in one of it's many excellent debugging modes or you can simply configure a remote email client to use ASMTP. This guide will use the commandline to test ASMTP.
Important:
Before attempting this method please read the exim documentation on how ASMTP works. The following assumes you have read and understood that text.
First create a simple perl script called 'encode' in /usr/local/etc/exim/ and make sure it is executable:
CODE:
root@win /usr/local/etc/exim# cat encode
#!/usr/bin/perl
use MIME::Base64;
printf ("%s", encode_base64(eval ""$ARGV[0]""));
root@darkstar /usr/local/etc/exim# chmod +x encode
root@darkstar /usr/local/etc/exim# ls -al encode
-rwxr-xr-x 1 root wheel 85 Apr 23 12:25 encode
#!/usr/bin/perl
use MIME::Base64;
printf ("%s", encode_base64(eval ""$ARGV[0]""));
root@darkstar /usr/local/etc/exim# chmod +x encode
root@darkstar /usr/local/etc/exim# ls -al encode
-rwxr-xr-x 1 root wheel 85 Apr 23 12:25 encode
Now decide which user account on your server you wish to test ASMTP with. It must be an account you know the password for obviously. I created an account called 'dummy' and set the password to 'dummy' as well - if you do this remember to remove the account or disable it as soon as you've finished testing.
Encode the user:password pair into base64 MIME using the 'encode' script we created above:
CODE:
root@darkstar /usr/local/etc/exim# ./encode "\0dummy\0dummy"
AGR1bW15AGR1bW15
AGR1bW15AGR1bW15
Now enter into Exim's fake SMTP session command-line mode and just for good measure do it in authentication debug mode as well:
CODE:
root@win /root# exim -d+auth -bh 127.0.0.1
Exim version 4.66 (FreeBSD 6.1) uid=0 gid=0 pid=3056 D=fbb95cfd
Probably Berkeley DB version 1.8x (native mode)
Support for: crypteq iconv() IPv6 use_setclassresources PAM Perl OpenSSL Content_Scanning Old_Demime
Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch nis nis0 passwd
Authenticators: cram_md5 plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Size of off_t: 8
changed uid/gid: forcing real = effective
uid=0 gid=0 pid=3056
auxiliary group list: 0
seeking password data for user "mailnull": using cached result
getpwnam() succeeded uid=26 gid=26
seeking password data for user "root": cache not available
getpwnam() succeeded uid=0 gid=0
configuration file is /usr/local/etc/exim/configure
log selectors = 00000ffc 00089001
trusted user
admin user
changed uid/gid: privilege not needed
uid=26 gid=6 pid=3056
auxiliary group list: 6 6
seeking password data for user "mailnull": cache not available
getpwnam() succeeded uid=26 gid=26
originator: uid=0 gid=0 login=root name=Charlie Root
sender address = root@win.munk.me.uk
sender_fullhost = [127.0.0.1]
sender_rcvhost = [127.0.0.1]
**** SMTP testing session as if from host 127.0.0.1
**** but without any ident (RFC 1413) callback.
**** This is not for real!
host in hosts_connection_nolog? no (option unset)
LOG: smtp_connection MAIN
SMTP connection from [127.0.0.1]
host in host_lookup? yes (matched "*")
looking up host name for 127.0.0.1
DNS lookup of 1.0.0.127.in-addr.arpa (PTR) succeeded
IP address lookup yielded localhost.munk.me.uk
gethostbyname2 looked up these IP addresses:
name=localhost.munk.me.uk address=::1
name=localhost.munk.me.uk address=127.0.0.1
checking addresses for localhost.munk.me.uk
::1
127.0.0.1 OK
sender_fullhost = localhost.munk.me.uk [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1])
set_process_info: 3056 handling incoming connection from localhost.munk.me.uk [127.0.0.1]
host in host_reject_connection? no (option unset)
host in sender_unqualified_hosts? no (option unset)
host in recipient_unqualified_hosts? no (option unset)
host in helo_verify_hosts? no (option unset)
host in helo_try_verify_hosts? no (option unset)
host in helo_accept_junk_hosts? no (option unset)
SMTP>> 220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
smtp_setup_msg entered
Exim version 4.66 (FreeBSD 6.1) uid=0 gid=0 pid=3056 D=fbb95cfd
Probably Berkeley DB version 1.8x (native mode)
Support for: crypteq iconv() IPv6 use_setclassresources PAM Perl OpenSSL Content_Scanning Old_Demime
Lookups: lsearch wildlsearch nwildlsearch iplsearch cdb dbm dbmnz dnsdb dsearch nis nis0 passwd
Authenticators: cram_md5 plaintext spa
Routers: accept dnslookup ipliteral manualroute queryprogram redirect
Transports: appendfile/maildir/mailstore/mbx autoreply lmtp pipe smtp
Fixed never_users: 0
Size of off_t: 8
changed uid/gid: forcing real = effective
uid=0 gid=0 pid=3056
auxiliary group list: 0
seeking password data for user "mailnull": using cached result
getpwnam() succeeded uid=26 gid=26
seeking password data for user "root": cache not available
getpwnam() succeeded uid=0 gid=0
configuration file is /usr/local/etc/exim/configure
log selectors = 00000ffc 00089001
trusted user
admin user
changed uid/gid: privilege not needed
uid=26 gid=6 pid=3056
auxiliary group list: 6 6
seeking password data for user "mailnull": cache not available
getpwnam() succeeded uid=26 gid=26
originator: uid=0 gid=0 login=root name=Charlie Root
sender address = root@win.munk.me.uk
sender_fullhost = [127.0.0.1]
sender_rcvhost = [127.0.0.1]
**** SMTP testing session as if from host 127.0.0.1
**** but without any ident (RFC 1413) callback.
**** This is not for real!
host in hosts_connection_nolog? no (option unset)
LOG: smtp_connection MAIN
SMTP connection from [127.0.0.1]
host in host_lookup? yes (matched "*")
looking up host name for 127.0.0.1
DNS lookup of 1.0.0.127.in-addr.arpa (PTR) succeeded
IP address lookup yielded localhost.munk.me.uk
gethostbyname2 looked up these IP addresses:
name=localhost.munk.me.uk address=::1
name=localhost.munk.me.uk address=127.0.0.1
checking addresses for localhost.munk.me.uk
::1
127.0.0.1 OK
sender_fullhost = localhost.munk.me.uk [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1])
set_process_info: 3056 handling incoming connection from localhost.munk.me.uk [127.0.0.1]
host in host_reject_connection? no (option unset)
host in sender_unqualified_hosts? no (option unset)
host in recipient_unqualified_hosts? no (option unset)
host in helo_verify_hosts? no (option unset)
host in helo_try_verify_hosts? no (option unset)
host in helo_accept_junk_hosts? no (option unset)
SMTP>> 220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
220 win.munk.me.uk ESMTP Exim 4.66 Wed, 17 Jan 2007 19:24:22 +0000
smtp_setup_msg entered
When you get to this point you are ready to start an SMTP 'conversation' with Exim. First introduce yourself to Exim using the SMTP 'EHLO localhost' command:
CODE:
EHLO localhost
SMTP<< EHLO localhost
sender_fullhost = localhost.munk.me.uk (localhost) [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1] helo=localhost)
set_process_info: 3103 handling incoming connection from localhost.munk.me.uk (localhost) [127.0.0.1]
host in pipelining_advertise_hosts? yes (matched "*")
host in auth_advertise_hosts? yes (matched "*")
host in tls_advertise_hosts? no (option unset)
250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
SMTP>> 250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
SMTP<< EHLO localhost
sender_fullhost = localhost.munk.me.uk (localhost) [127.0.0.1]
sender_rcvhost = localhost.munk.me.uk ([127.0.0.1] helo=localhost)
set_process_info: 3103 handling incoming connection from localhost.munk.me.uk (localhost) [127.0.0.1]
host in pipelining_advertise_hosts? yes (matched "*")
host in auth_advertise_hosts? yes (matched "*")
host in tls_advertise_hosts? no (option unset)
250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
SMTP>> 250-win.munk.me.uk Hello localhost.munk.me.uk [127.0.0.1]
250-SIZE 52428800
250-PIPELINING
250-AUTH PLAIN LOGIN
250 HELP
In response to your 'EHLO localhost' command, Exim returns more debug information but most importantly for us it also indicates what authentication options it offers in this line:
CODE:
250-AUTH PLAIN LOGIN
This indicates that currently acceptable AUTH methods are PLAIN and LOGIN.
We can then test the PLAIN login method using the "\0dummy\0dummy" user:password pair we encoded above:
CODE:
AUTH PLAIN AGR1bW15AGR1bW15
SMTP<< AUTH PLAIN AGR1bW15AGR1bW15
Running pwcheck authentication for user "dummy"
pwcheck: success (NULL)
plain authenticator:
$1 =
$2 = dummy
$3 = dummy
expanded string: 1
SMTP>> 235 Authentication succeeded
235 Authentication succeeded
SMTP<< AUTH PLAIN AGR1bW15AGR1bW15
Running pwcheck authentication for user "dummy"
pwcheck: success (NULL)
plain authenticator:
$1 =
$2 = dummy
$3 = dummy
expanded string: 1
SMTP>> 235 Authentication succeeded
235 Authentication succeeded
This indicates that authentication for 'dummy:dummy' would succeed and mail would be relayed (pending further conditional checks by Exim).
So we now have a working Exim with support for spam/malware checking and authentication over SMTP.
Sunday, December 24. 2006
Portupgrade fails to upgrade dependencies
When using portupgrade to upgrade ports recursively, occasionally you get a problem where portupgrade fails to upgrade a dependency of a port that's being upgraded. This seems to happen most often with perl ports - p5-* ports - probably because perl packages/ports use the most dependencies of all with being modular in design anyway.
An example is when I just went to run the weekly 'portupgrade -arR' and whilst upgrading p5-PathTools-3.21, portupgrade found that another port - p5-Scalar-List-Utils-1.18 - also needed upgrading. Unfortunately the upgrade of that port failed with the following error:
The problem is that any already installed dependencies - regardless of whether they need upgrading or not - are seen by portupgrade as being installed already and so it refuses to upgrade. Hence if the port you're trying to upgrade has a dependency that *also* needs upgrading, this will fail. Adding '-f' onto the portupgrade line makes no difference.
One solution is to set the environment variable 'FORCE_PKG_REGISTER':
and then run the portupgrade command again. The installed status of the dependencies are then effectively ignored and the port dependencies are forcibly installed. Not sure why portupgrade doesn't upgrade any dependencies automatically - I would have thought any dependencies of a port would be automatically checked for upgrades and upgraded if necessary. Maybe I'm missing something.
An example is when I just went to run the weekly 'portupgrade -arR' and whilst upgrading p5-PathTools-3.21, portupgrade found that another port - p5-Scalar-List-Utils-1.18 - also needed upgrading. Unfortunately the upgrade of that port failed with the following error:
CODE:
===> Checking if lang/p5-Scalar-List-Utils already installed
===> p5-Scalar-List-Utils-1.18,1 is already installed
You may wish to ``make deinstall'' and install this port again
by ``make reinstall'' to upgrade it properly.
If you really wish to overwrite the old port of lang/p5-Scalar-List-Utils
without deleting it first, set the variable "FORCE_PKG_REGISTER"
in your environment or the "make install" command line.
*** Error code 1
Stop in /home/munk/ports/lang/p5-Scalar-List-Utils.
*** Error code 1
Stop in /home/munk/ports/devel/p5-PathTools.
===> p5-Scalar-List-Utils-1.18,1 is already installed
You may wish to ``make deinstall'' and install this port again
by ``make reinstall'' to upgrade it properly.
If you really wish to overwrite the old port of lang/p5-Scalar-List-Utils
without deleting it first, set the variable "FORCE_PKG_REGISTER"
in your environment or the "make install" command line.
*** Error code 1
Stop in /home/munk/ports/lang/p5-Scalar-List-Utils.
*** Error code 1
Stop in /home/munk/ports/devel/p5-PathTools.
The problem is that any already installed dependencies - regardless of whether they need upgrading or not - are seen by portupgrade as being installed already and so it refuses to upgrade. Hence if the port you're trying to upgrade has a dependency that *also* needs upgrading, this will fail. Adding '-f' onto the portupgrade line makes no difference.
One solution is to set the environment variable 'FORCE_PKG_REGISTER':
CODE:
setenv FORCE_PKG_REGISTER 1
and then run the portupgrade command again. The installed status of the dependencies are then effectively ignored and the port dependencies are forcibly installed. Not sure why portupgrade doesn't upgrade any dependencies automatically - I would have thought any dependencies of a port would be automatically checked for upgrades and upgraded if necessary. Maybe I'm missing something.
Saturday, September 30. 2006
Stunnel upgrade causes portupgrade to stall
I usually run 'portupgrade -arR' once a week and leave it to itself to get on with it. Unfortunately sometimes I come back and find only a couple of ports have been upgraded and portupgrade has stalled waiting for me to enter some input about something or other. The usual suspect is the upgrade of stunnel - stunnel asks you at the deinstallation stage whether it should remove the 'stunnel' user and group and the portupgrade won't proceed until you've answered yes or no.
This is all well and good if you're permanantly deleting the port, you don't really want unnecessary users and groups left on your system. However in the case of portupgrade, the user/group will be used again so asking if you want to remove the user/group doesn't really need an answer - just leave the user/group alone and get on with it already!
I had a quick google for 'stunnel portupgrade user groups' and found someone who'd had the same problem. The solution the guy mentioned is to set the option PACKAGE_BUILDING for the stunnel port in the portupgrade config file - /usr/local/etc/pkgtools.conf:
Unfortunately as yet this doesn't seem to work - neither does setting 'BATCH' which usually does the trick anyway (the guy complaining about hte issue noted that BATCH didn't work for him, whist PACKAGE_BUILDING did. Neither works for me :(). Anyway I'll probably spend a few minutes looking into it and see if I can sort it... in the mean time if anyone has any ideas let me know in the comments :)
UPDATE:
Mmm ok, had a quick look into the problem and it looks like portupgrade isn't taking any of the flags set in pkgtools.conf - at the start of the upgrade it says 'BATCH=1' is set, but when the deinstall script runs the value of BATCH is reset to empty for some reason.
UPDATE2: I think the problem is because whilst portupgrade honours the env variables set in pkgtools.conf, the pkg-deinstall script for stunnel doesn't inherit any env variables because it's invoked from pkg_delete. The only solution I can think of off top of head is to export the BATCH variable on the commandline along the lines of:
setenv BATCH 1
just before running portupgrade. PITA (works though, just tested it:).
This is all well and good if you're permanantly deleting the port, you don't really want unnecessary users and groups left on your system. However in the case of portupgrade, the user/group will be used again so asking if you want to remove the user/group doesn't really need an answer - just leave the user/group alone and get on with it already!
I had a quick google for 'stunnel portupgrade user groups' and found someone who'd had the same problem. The solution the guy mentioned is to set the option PACKAGE_BUILDING for the stunnel port in the portupgrade config file - /usr/local/etc/pkgtools.conf:
CODE:
'security/stunnel' => [ 'PACKAGE_BUILDING=1' ],
Unfortunately as yet this doesn't seem to work - neither does setting 'BATCH' which usually does the trick anyway (the guy complaining about hte issue noted that BATCH didn't work for him, whist PACKAGE_BUILDING did. Neither works for me :(). Anyway I'll probably spend a few minutes looking into it and see if I can sort it... in the mean time if anyone has any ideas let me know in the comments :)
UPDATE:
Mmm ok, had a quick look into the problem and it looks like portupgrade isn't taking any of the flags set in pkgtools.conf - at the start of the upgrade it says 'BATCH=1' is set, but when the deinstall script runs the value of BATCH is reset to empty for some reason.
UPDATE2: I think the problem is because whilst portupgrade honours the env variables set in pkgtools.conf, the pkg-deinstall script for stunnel doesn't inherit any env variables because it's invoked from pkg_delete. The only solution I can think of off top of head is to export the BATCH variable on the commandline along the lines of:
setenv BATCH 1
just before running portupgrade. PITA (works though, just tested it:).
Saturday, September 16. 2006
Strategy for Upgrading Ports
Just read a great article called Top ten rules for using ports on BSDPants - BSDPants looks like a good blog btw if you're into FreeBSD, check it out. Got me to thinking about my strategy for upgrading my ports.
My port upgrade strategy - in order of prioritization - is:
Freshports Daily Watch List Notifications
I should also mention the excellent watch list service at freshports.org - you create a watch list of all the ports you currently have on your system and then opt to get daily email notification when any of those ports are updated. This is a very handy thing in combination with the strategy above - the notification email includes the cvs commit message for the ports that have changed so you know how urgent the update is and whether you should do something about it immediately or let it wait for the weekly update.
Read the FAQ article linked in the previous paragraph for more info on watch lists - easiest thing is to just register on the freshports.org site and get stuck in though.
I've been meaning to write a script to update my watch list at freshports automatically - it will get done eventually and I'll post about it here when it happens.
I'll include the cvsup_cron.sh script in the extended article, it requires the 'fastest_cvsup' port to be installed - a useful script for working out which cvsup server is fastest/closest in terms of ping speeds, worth installing if you haven't got it already.
My port upgrade strategy - in order of prioritization - is:
- Get a daily portaudit report of any ports that have security problems. Upgrade those ports as soon as I get the report.
Portaudit is a tool that checks each port you have installed against a list of ports that have security problems. If any of your ports match up with the ports on the security list, they're flagged as being insecure and you receive a notification of this when the periodic script is run.
On a daily basis, have a script - cvsup_cron.sh - run the following. I'll add the script in the extended body, but essentially all it does is:
CODE:
# cvsup the ports tree to latest ver:
/usr/local/bin/cvsup -g -L 0 /etc/supfile-ports-all
# Use portupgrade to fetch any distfiles for any ports that have been updated:
/usr/local/sbin/portupgrade -arRF
# Then output a list of all those ports:
/usr/local/sbin/portversion -vl "<"
/usr/local/bin/cvsup -g -L 0 /etc/supfile-ports-all
# Use portupgrade to fetch any distfiles for any ports that have been updated:
/usr/local/sbin/portupgrade -arRF
# Then output a list of all those ports:
/usr/local/sbin/portversion -vl "<"
This updates my ports tree to the latest, fetches any distfiles for newly updated ports and then prints out a list of those ports that are out of date. Fetching the updated distfiles just makes upgrading faster when I'm sat there in front of the console doing the work - instead of waiting ages for distfiles to download, the distfiles are already there waiting to be installed.
From the daily cvsup_cron.sh script output, if there are any services/servers that need updating, I'll update those immediately. Why? Well on a Saturday I upgrade all the out of date ports in one go using the '-arR' flags with portupgrade. This recursively upgrades all the out of date ports and those that rely on them, but if any of those are services then there's a chance that the service will be taken down by the upgrade process. By upgrading any services seperately I can make sure the services are restarted immediately instead of waiting for the 'portupgrade -arR' process to finish (I usually go off and do something else anyway whilst that's going on:).
Finally on a Saturday as mentioned above, I upgrade every port that's out of date automatically using 'portupgrade -arR'. I like to try and read /usr/ports/UPDATING first for any info on what might go wrong, but I have to admit I don't usually read it until something's already gone wrong - doh.
Freshports Daily Watch List Notifications
I should also mention the excellent watch list service at freshports.org - you create a watch list of all the ports you currently have on your system and then opt to get daily email notification when any of those ports are updated. This is a very handy thing in combination with the strategy above - the notification email includes the cvs commit message for the ports that have changed so you know how urgent the update is and whether you should do something about it immediately or let it wait for the weekly update.
Read the FAQ article linked in the previous paragraph for more info on watch lists - easiest thing is to just register on the freshports.org site and get stuck in though.
I've been meaning to write a script to update my watch list at freshports automatically - it will get done eventually and I'll post about it here when it happens.
I'll include the cvsup_cron.sh script in the extended article, it requires the 'fastest_cvsup' port to be installed - a useful script for working out which cvsup server is fastest/closest in terms of ping speeds, worth installing if you haven't got it already.
Continue reading "Strategy for Upgrading Ports"
(Page 1 of 3, totaling 11 entries)
» next page