SSH

For a long time now I've had a lot of problems with brute force attacks against sshd and proftpd - attacks where a host will attempt to login with a dictionary of common usernames and passwords, trying each one until they find a combination that works. Apart from being a security issue, this uses up a lot of bandwidth so it's worth taking some measures to block these kind of attacks.

Both sshd and ftpd services have their own individual means for blocking individual connections, but unfortunately neither have an inbuilt method for detecting brute force attacks - counting how many failed login attempts are made from each individual IP address and then blocking that IP address if the number of failed login attempts is more than a certain number. This is where a 3rd party utility is required.

There are a few utilities that can mitigate brute force attacks on services. For a while now I've used DenyHosts successfully to block sshd brute force attacks. DenyHosts works by constantly monitoring sshd logfiles and keeping track of how many failed logins have occured per IP address over time. If the number of failed logins reaches a certain threshold, DenyHosts adds an entry in /etc/hosts.allow that effectively blocks the IP address, stopping that host from connecting to the sshd service any more.

DenyHosts is great, but unfortunately it's aimed only at blocking sshd brute force attacks and I need to protect the ftpd service as well as just sshd - and in future maybe adapt to block other services. With this in mind I decided to move to using a very similar script called BlockHosts (the documentation for BlockHosts actually mentions that it was inspired by DenyHosts). BlockHosts can scan a list of service logfiles in one go instead of just a single logfile as with DenyHosts, so is ideal for monitoring a number of different services for brute force attacks.

The following describes how to install and configure BlockHosts on FreeBSD so it's executed every time the sshd or proftpd services are accessed using TCP_WRAPPERS - ie modifying /etc/hosts.allow so the blockhosts script is run each time sshd or proftpd are accessed. The BlockHosts script will then check if this current connection attempt is part of a brute force attack and if so, add a blocking rule to /etc/hosts.allow to deny further access.

Installation of BlockHosts

1. Download blockhosts from the download page, extract the distribution (note please check the download link for the latest version, the version below was latest at time of writing):
   
   CODE:
   root@users /home/munk/bin/python/blockhosts# wget http://www.aczoom.com/tools/blockhosts/BlockHosts-1.0.5.tar.gz
   root@users /home/munk/bin/python/blockhosts# tar zxvf BlockHosts-1.0.5.tar.gz
   BlockHosts-1.0.5/
   BlockHosts-1.0.5/Makefile
   ...
   
   
   
2. Change to BlockHosts directory:
   
   CODE:
   root@users /home/munk/bin/python/blockhosts# cd BlockHosts-1.0.5
   
   
   
3. Edit and save blockhosts.py to read:
   
   
   CODE:
   CONFIG_FILE = "/usr/local/etc/blockhosts.cfg"
   ...
       "LOGFILES": ( "/var/log/auth.log", ),
   
   Note: may seem a bit odd editing the blockhosts.py script before it's installed - the reason for this is that the installation locations used by setup.py below are taken from blockhosts.py, so by modifying blockhosts.py like this we get the config file installed into /usr/local/etc/ (FreeBSD default for 3rd party software) instead of into /etc (default for linux 3rd party software).
   
   
4. Install blockhosts:
   
   
   CODE:
   root@users /home/munk/bin/python/blockhosts/BlockHosts-1.0.5# python setup.py -v install
   running install
   running build
   running build_scripts
   creating build
   creating build/scripts-2.4
   copying and adjusting blockhosts.py -> build/scripts-2.4
   changing mode of build/scripts-2.4/blockhosts.py from 644 to 755
   running install_scripts
   copying build/scripts-2.4/blockhosts.py -> /usr/local/bin
   changing mode of /usr/local/bin/blockhosts.py to 755
   running install_data
   copying blockhosts.cfg -> /usr/local/etc
   
   
   This installs the blockhosts.py script into /usr/local/bin and the config file blockhosts.cfg into /usr/local/etc. Make sure to run 'rehash' to reread the binary paths again so blockhosts.py will run from anywhere:
   
   
   CODE:
   root@users /home/munk/bin/python/blockhosts/BlockHosts-1.0.5# rehash
   
   
   
   
5. Edit and save the /usr/local/etc/blockhosts.cfg file so it reads:
   
   
   CODE:
   LOGFILES = [ "/var/log/auth.log", "/var/log/ftp.log" ]
   
   
   Important:
   Add the logfiles you want blockhosts to monitor for brute force attacks here. /var/log/auth.log is standard for sshd, /var/log/ftp.log is maybe not standard for all ftpd, this is just what I have setup here.
   
   At this point it's best to read through the documentation for blockhosts completely - the README, INSTALL and the blockhosts.py script itself. The following section is pretty much copy/pasted from what's mentioned in there.
   
   
   
6. Edit and save /etc/hosts.allow to include the section that blockhosts.py will modify. Make sure you allow your own IP blocks first and any trusted IPs so they don't get blocked accidentally:
   
   GOTCHA LOOKOUT!
   One gotcha to watch out for in this is the line:
   
   
   CODE:
   ALL : ALL : allow
   
   
   You MUST remove this line - replace it with your IP block instead so you don't get locked out from your own address range. If this line isn't removed/commented out, anything below it just isn't read/executed and blockhosts won't work.
   
   This is how my /etc/hosts.allow looks:
   
   
   CODE:
   #######################################################################
   # blockhosts
   #######################################################################
   # ----
   # see "man 5 hosts_access" for details of the format of IP addresses,
   #services, allow/deny options. Also see "man hosts_options"
   #order of lines in this file is important, first matched IP address line
   #is rule applied by hosts_access
   #
   # permanent whitelist addresses - these should always be allowed access
   
   ALL : 213.152.51.192/255.255.255.248 : allow
   # ALL: 127.0.0.1  : allow
   # ALL: 192.168.0. : allow
   
   # permanent blacklist addresses - these should always be denied access
   
   # ALL: 10.  : deny
   # ALL: 192. : deny
   # ALL: 172. : deny
   
   # ----------------------------------------
   # next section is the blockhosts section - it will add/delete entries in
   # between the two marker lines (#---- BlockHosts Additions)
   
   #---- BlockHosts Additions
   #---- BlockHosts Additions
   
   # ----------------------------------------
   # finally, the command to execute the blockhosts script, based on
   # connection to particular service or services, for example, for
   # sshd and proftpd - if using vsftpd, pure-ftpd, be sure to use those
   # words instead:
   
   sshd, proftpd: ALL: spawn (/usr/local/bin/blockhosts.py --verbose --echo "%c-%s" >> /var/log/blockhosts.log 2>&1 )& : allow
   
   # remove:   >> /var/log/blockhosts.log 2>&1     if logging to
   # blockhosts.log is not needed - it will still log to syslog (minimally)
   # see examples below
   # --
   # See "man hosts.allow" for info on %c and %s identifiers
   #----
   # for non-verbose, with identification, to syslog only (/var/log/messages):
   #sshd, proftpd, in.proftpd: ALL: spawn /usr/bin/blockhosts.py --echo "%c-%s" & : allow
   #----
   # minimal logging, to syslog (usually goes to /var/log/messages):
   #sshd, proftpd, in.proftpd: ALL: spawn /usr/bin/blockhosts.py & : allow
   #----
   # To test hosts.allow, and to find out exact names of SSH/FTP services,
   # add this line to the beginning of hosts.allow, use ssh/ftp to connect
   # to your server, and then look at the log (/var/log/messages or
   # blockhosts.log) to see the name of the invoked service.
   # IMPORTANT: after your test is done, remove this line from hosts.allow!
   # Otherwise everyone will always have access.
   #ALL : ALL: spawn (/usr/bin/blockhosts.py --verbose --echo "%c-%s" >> /var/log/blockhosts.log 2>&1 )& : allow
   #######################################################################
   # blockhosts
   #######################################################################
   
   
   Important Note for ProFTPD users:
   The following sections describes the configuration needed when using proftpd via inetd. If you are using ProFTPD in standalone mode, you need to use the proftpd mod_wrap/mod_wrap_file functionality to have proftpd read and honour the TCP_WRAPPERS//etc/hosts.allow file(s) when denying/allowing hosts. Additionally you need to specifiy the configure flag --enable-wrapper-options when building proftpd. For a heavily used server, this might be worth doing but personally I don't get that many connections that I need to worry about inetd being overloaded so I can just go down the (easier to configure for blockhosts) inetd path.
   
   
7. Ensure proftpd is configured to run correctly via inetd.
   
   Edit and save /usr/local/etc/proftpd.conf to read:
   
   CODE:
   ServerType          inetd
   
   
   Important: remember to delete or rename /usr/local/etc/rc.d/proftpd.sh so it's not run at boot time - the proftpd daemon doesn't need to be started at boot if you're using inetd, inetd handles all the proftpd connections, see below:
   
   Edit and save /etc/inetd.conf to read:
   
   CODE:
   ftp stream  tcp nowait  root    /usr/local/sbin/in.proftpd proftpd
   
   
   then restart inetd:
   
   
   CODE:
   root@users /usr/local/etc# kill -HUP `cat /var/run/inetd.pid `
   
   This forces inetd to restart, rereading the config file changes made to /etc/inetd.conf. ftp connections will now be handled by proftpd via inetd.
   

We're now ready to run blockhosts.py for the first time. BlockHosts will parse each logfile mentioned in blockhosts.cfg and check for any brute force attacks and if it finds any, blocks will be added to the /etc/hosts.allow file.

Note: This initial check does not take into account the period over which failed logins took place, so any IP that has more than the default 7 failed login entries will look like a brute force attacker. However, the ban BlockHosts adds will only last for the default 12 hours so this shouldn't cause a huge issue - just be aware of this and check the IPs that are added on the first run.

For the very first time it's a good idea to try a 'dry run' just to see what blockhosts finds and what it'd do, without actually doing anything to the /etc/hosts.allow file. To do this, run blockhosts with the '--dry-run' flag:



From this you can see nicely what blockhosts makes of the service logfiles and the addresses that have tried to connect unsuccessfully. On my host, as you can see above, there are a few that are obviously dodgy (I would only expect a max of maybe 8 connections per ip per month, so clearly 29,499 connections is just wrong!).

Once you're happy that the output is correct, run blockhosts again without the '--dry-run' flag and the /etc/hosts.allow file will be modified. Also from now on the logfiles will only be read from the last recorded offset which saves a lot of time if your logfiles are very big.

Big thanks to the BlockHosts author Avinash Chopde !

The locate utility on linux was one of the first tools I hit when I made the move to FreeBSD a few years back - knowing where files are is half the battle when you're trying to configure things and find documentation on how to do it. The trouble with locate though as jdarnold mentions in his article 'Locate This!' is that if you build the locate database as 'root', you end up exposing everything to any user that runs the locate command. The other problem he mentions is the locate db is only updated weekly on FreeBSD by default via the periodic system which isn't really enough if you use your system regularly.

I remember thinking along the same lines a while back and after reading through the man pages the solution I found was to create two separate databases - one for root and one for regular users. The 'regular' db is updated on a weekly basis as per the default on FreeBSD via periodic, whereas the other 'root' locate db is built daily in a crontab so I can get the latest up to date details on which files are where.

To get the root db built first you need to create a crontab entry - i put this in /etc/crontab:



This tells the locate.updatedb script to use a separate configuration file - /root/locate/conf/locate.rc - for building root's locate db. The content of /root/locate/conf/locate.rc look like this:



which indicates that this db should be built in /root/locate/db/locate.database.root instead of the default locate in /var/db/locate.database. You can safely run the command as root on the commandline to initialize your new db:



Once the database is built you can move on to test the new db works ok:



This file is only readable by root, so it seems to work ok.


To make things easier, add a shell alias in root's .cshrc file aliasing 'locate' to the command 'locate -d /root/locate/db/locate.database.root':



With the "-d /root/locate/db/locate.database.root" switch, locate will use the db at /root/locate/db/locate.database.root instead of the default /var/db/locate.database and root will be able to use locate to find any files in the filesystem, not just those that are world readable.


Finally, one way to update the regular locate db as root but without making it list every world readable file is to perform the following:



This is basically what the 310.locate periodic script does and results in a locate db that contains only files that are readable by the 'nobody' user - essentially all 'world readable' files.

Comparing the sizes of the root db against the nobody db:



You can see the size difference there, not as many entries in nobody's db as root's. Just to double check:



So from that you can see that 'nobody' can see the ktrace.out files located in /root - apart from root of course :) Sorted.

parsepath.pl is a brilliant perl script for fixing permissions problems on Unix based platforms by Jeremy Mates. Probably the most common type of permission problem from a sysadmin/webmaster's viewpoint is uploading a file to a directory in a website's document root folder and then trying to access the file or script in a web browser only to get the dreaded 403 error message:

Forbidden
You don't have permission to access /foo/bar/test.php on this server.

Most time the solution is very simple, just change the permissions on 'test.php' to make sure the user the webserver runs as can read the file correctly - the simplest and most common method being to change the mode of the file to '755':



Unfortunately sometimes it's not that easy and many times you see users asking 'I'm getting 'access denied' errors even though I've changed the perms to 755'. The problem is that one of the subdirectories that the 'test.php' file lives in has permissions set so that the webserver can't read the file properly. Now that's where the headache comes in :)

However, parsepath.pl can take the headache out of fixing permissions problems.

Say you have a website document root directory tree /usr/local/www/web/www.munk.me.uk/foo/bar and you upload a web script 'test.php' into that directory. You try and access the file in a webbrowser but get the 403 permission denied error above. First off you check the permissions on the file itself:



That looks ok, with permissions 755 and the owner/group set to 'www' the webserver user 'www' should be able to read the file ok. So in this case the problem must be with the permissions on one of the parent subdirectories. The old method of working out the perms would be either to trawl one by one through each directory checking the perms on each subdirectory or to change the permissions recursively on the document root folder so all subfolders have the read bit set for the webserver user/group.

With parsepath.pl things are a lot simpler though - just run the following command:



With this command parsepath.pl recurses through each subdirectory below the file/path you feed it on the commandline and tells you the permissions problems - if any - for the user 'www' (the user=www argument) to read (the +r argument) the file 'test.php'.

In the output, we're told that permissions to read the test.php by the user www fails on two counts:



With this information it's easy enough to go in and make the changes necessary to fix the problem using 'chmod g+rx foo foo/bar'.

There are other ways of invoking parsepath.pl though. Running it just with a file/path as an argument it'll tell you the permissions on each subdirectory under it:



which can is better to see a whole tree in one go.

No permissions were harmed in the making of this article!


I'll include the parsepath.pl script in the extended article just in case the original ever gets lost - big credit of course goes to the author of the script, Jeremy Mates. His site is actually very interesting from a sysadmin's point of view containing lots of interesting admin scripts and thoughts on system administration in general - spent quite a while grazing through his stuff there - cheers Jeremy.

I use the SSH client Putty to login to my FreeBSD server from my Windows workstation and entering passwords on the commandline can be a chore. Starting up Putty and connecting to servers you use regularly via SSH can be made a lot easier though by using secure keys, allowing you to fire up putty and get 'auto logged in' without the need for password entry.

The idea is you create a pair of keys on the machine from which you'll be logging in via SSH, one public and one private. The public key is stored on the server to which you're logging into and the private key is stored at the machine you'll be logging in from. Once both keys are in place, you can associate your private key with the SSH session associated with the server where the public key lives at the other end. Now when you login you don't need to enter a password - Putty sends the public key information along at the authentication phase, the remote server checks the private key against the public key it finds on the server and if everything checks out you're logged in without being prompted for a password.

There's a great article for setting up a key pair in the PuTTYgen utility and installing the key on the remote server on this site, the basic steps are:

1. Run the PuTTYgen key pair generator to create a public and private key - PuTTYgen can be found here on the Putty site. Note if you don't want to have to enter any password at all (see security disclaimer below!), do not enter a passphrase to protect the private key
   
2. Copy the public key into the file ~/.ssh/authorized_keys (or ~/.ssh/authorized_keys2 for SSH2, though I think SSH2 still reads ~/.ssh/authorized_keys). The public key usually looks similar to this:
   
   CODE:
   ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEApiVHreNg3Xnmrl9lo6nCpsb+
   iytNHBZDztLshEhLaucQCs66b0K2fjpfl3KheQB9lpIBBtL0NaPZslo1fVGnJT8fF4ZywIyNuKzN8cYM+
   zfIMjqe8UxQD1QfE1s4QxBhgyihGb6PwQkYJeWfMaINq5pnuHzx2Fc1kWRMdaGQvSU= munk@winny
   (actually broken up so it displays properly here, usually a single line)
   
   
3. Ensure the ~/.ssh/authorized_keys file is readable only by the user:
   
   CODE:
   chmod 600 ~/.ssh/authorized_keys
   
   
4. Finally, tell Putty to use the private key for all SSH sessions with the remote server. This is done by selecting the session settings in putty for the server that has the public key, clicking 'Load', selecting the 'Connection, SSH, Auth' setting page and then selecting the private key file on that page using the Browse button. Save the settings by going back to the 'Session' page tab and hitting 'Save'.

If all went well and the private key was created without using a password phrase, this should allow you to login to the remote server without the need for a password.

However IMPORTANTLY it should be stressed for obvious reasons using no passphrase to protect the private key should only be contemplated on a machine on which you know noone else uses or has access to - either physically or across a network. I feel like a dirty security whore in a way for even suggesting not using a security passphrase for the private key, but the truth of it is noone has physical access to this machine who would care about accessing my servers so ...

I'm off to setup a passphrase now...

:)