freebsd.munk.me.uk - Shell

Command Line Web Browsing With WWW::Mechanize::Shell

nospam@example.com (munk) — Wed, 15 Aug 2007 12:46:00 +0000

Introduction
The perl module WWW::Mechanize::Shell is a brilliant tool for browsing websites at a very low level - think somewhere in between using telnet and using a command line based browser like lynx or links or w3m and you'll be close. WWW::Mechanize::Shell is more than that though, it allows you to script a complete HTTP session so it can be replayed back at a later date without any interaction using WWW::Mechanize::Shell's parent perl module WWW::Mechanize - great for automatically submitting HTML forms/ POST data regularly via a cron job for example.

In this article I'll be talking about installing WWW::Mechanize::Shell, look at a typical WWW::Mechanize::Shell browsing session and look at some examples of how I use WWW::Mechanize::Shell to make things easier. Finally the article will end with a real world example - using mechshell to automate logging into FreshPorts and updating a watch list.

Installing WWW::Mechanize::Shell
As the name suggests, WWW::Mechanize::Shell is a perl module whose 'parent' is the WWW::Mechanize module written by Andy Lester (WWW::Mechanize::Shell itself is written by Max Maischein at time of original writing). WWW::Mechanize does all the work in the background - WWW::Mechanize::Shell just makes it easy to interact in a HTTP session. WWW::Mechanize::Shell and all it's dependencies can be installed from the ports tree:

CODE:

root@users /root# cd /usr/ports/www/p5-WWW-Mechanize-Shell/
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# make install
root@users /home/munk/ports/www/p5-WWW-Mechanize-Shell# rehash

Getting Started Using WWW::Mechanize::Shell
Once installed, start up the WWW::Mechanize::Shell using the following you can use the following commandline:

CODE:

root@users /root# perl -MWWW::Mechanize::Shell -eshell

To make things easier though I use a CSH shell alias which aliases 'mechshell' to the command above:

CODE:

root@users /root# grep mechshell $cshrc
alias mechshell perl -MWWW::Mechanize::Shell -eshell

Examples of WWW::Mechanize Usage
I usually use WWW::Mechanize when I want to manipulate data from websites that require a stateful HTTP session - ie a browsing session where there's more than one URL you have to visit to complete the 'session'. Usually these kind of stateful sessions involve logging into the website first, then browsing to another page to obtain the data and then I have the WWW::Mechanize perl script handle the data and return any results on the commandline.

Some examples of scripts that I've use WWW::Mechanize with:

eclipse_flex_speed.pl
My ISP (Eclipse UK) used to allow you to 'flex' your internet speed from 256k up to 2Mbps. They ran an offer for a while where you could flex to the max for 3 months - unfortunately you could only flex for 12 hours at a time, which meant logging into the control panel every 12hrs, selecting the maximum speed and then submitting the form. PITA basically.

Instead I wrote eclipse_flex_speed.pl to automatically login to the Eclipse control panel, 'click' the 2Mbps radio button and then submit the form so my speed got flexed automagically. I then added the script as a cron job to autorun every 12hours, saving the haslle of doing it all manually!

aod_get.pl
The BBC website allows you to listen to streams of all BBC radio broadcasts for up to a week after they've been aired live. The problem is that the web interface you listen to the stream on in your web browser only allows you to skip 5 or 15 minutes ahead in time and doesn't allow you to go to specific times in the stream. To get around this you can obtain the URL of the real player stream and open it in a standalone real player - doing this you can go to any point in the stream easily. Trouble is finding the URL of the stream isn't that easy and involves viewing the source HTML of the web UI and copy/pasting a partial URL.

I started to write a WWW::Mechanize script to automate the 'screen scraping' of all the available feeds from the BBC Audio On Demand site and listing them on one single HTML page linking the name of the feed to the real player feed URL. As it goes though, someone else - Dave Cross - already had the same idea and wrote a great script for scraping the BBC feeds automatically. I now run this in a cronjob once a week.

torrentflux_ctl.pl
This is a script for starting and stopping all torrents under the control of the torrentflux web based bittorrent client. The script logs in as the torrent owner and then stops or starts all the torrents for that user - basically just does a GET of a URL that causes torrentflux to stop or start all torrents. Crude but effective.

Real world example - Automating the update of FreshPorts watch list
Below is a real world example usage of WWW::Mechanize::Shell - automating the procedure of updating your watch list on Freshports.org. I've included comments as '# this is a comment' to help explain what each command is doing.

CODE:

# Start up mechshell - alias for 'perl -MWWW::Mechanize::Shell -eshell':
munk@users /home/munk# mechshell

# Request the URL http://www.freshports.org/login.php.
# Note the HTTP response '(200)' is displayed underneath
# to indicate the page was fetched successfully:
(no url)>get http://www.freshports.org/login.php
Retrieving http://www.freshports.org/login.php(200)

# Use the mechshell 'dump' command to dump the contents
# of all forms found on the login page:
http://www.freshports.org/login.php>dump
POST http://www.freshports.org/login.php?origin=%2F [l]
 custom_settings=1 (hidden readonly)
 LOGIN=1 (hidden readonly)
 UserID= (text)
 Password= (password)
 submit=Login (submit)
 <NONAME>=<UNDEF> (reset)

# There's just a single form on this page:
# - the form's 'ACTION' is set to submit the form using the POST method
# to the url http://www.freshports.org/login.php?origin=%2F
# The form contains the following form fields:
# - 2 hidden fields
# - 1 text field called 'UserID'
# - 1 password field called 'Password'
# - 1 submit field called 'Login'

# Fill in the 'UserID' and 'Password' fields:
http://www.freshports.org/login.php>value UserID munk
http://www.freshports.org/login.php>value Password xxxxxx

# And then submit the form. Note we can just use the mechshell 'submit'
# command here because there is only a single form on the page. If there were
# more than one form on the page we would need to specify which button exactly to
# click:
http://www.freshports.org/login.php>submit
200

# Again note that the '200' response indicates the request was successful.
# Also note that the next mechshell prompt below has changed from
# 'http://www.freshports.org/login.php>' to just 'http://www.freshports.org/' -
# this indicates that the login script has probably redirected us to the
# freshports home page.

# Now we take a look to check that the login succeeded ok. To do this we use
# the mechshell 'content' command which effectively dumps the content of the
# returned page back at us in a pager.
# What we're looking for is the text 'Logged in as munk' which will indicate we
# logged in ok:
http://www.freshports.org/>content
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
-snip-
 <td NOWRAP>Logged in as munk <a href="http://freebsd.munk.me.uk/customize.php?origin=%2F" title="Customize your settings">Customize</a
> <a href="http://freebsd.munk.me.uk/logout.php" title="Logout of the website">Logout</a> <a href="/my-flagged-commits.php" title="Li
st of commits you have flagged">My Flagged Commits</a> 
-snip-

# Now we're logged in ok we can continue to upload the mypkg_info.txt file we
# created earlier.

# First browse to the pkg_upload.php page:
http://www.freshports.org/>get http://www.freshports.org/pkg_upload.php
Retrieving http://www.freshports.org/pkg_upload.php(200)

# Now use 'dump' to see a list of form fields on this page.
# Note that there are 2 submit buttons on this page:
http://www.freshports.org/pkg_upload.php>dump
POST http://www.freshports.org/pkg_upload.php (multipart/form-data)
 pkg_info= (file)
 staging=Staging (submit)
 wlid=5393 (option) [*5393/main*]
 replaceappend=replace (radio) [*replace/Replace list contents|append/Append to list (duplicates will be removed)]
 upload=Upload (submit)

# We need to fill out the form here. Uploading files with mechshell is as
# simple as completing the correct file type field:
http://www.freshports.org/pkg_upload.php>value pkg_info /tmp/mypkg_info.txt

# Ok, now we're ready to submit the form.
# Note that because there are 2 submit buttons on this form, we must explicitly
# tell mechshell which button it is that we want to click on - to do that we use
# the 'click' command. Just using 'submit' here would possibly click on the
# 'staging' button which is not what we want - instead we use the command
# 'click upload' to indicate we want to click on the 'upload' button:
http://www.freshports.org/pkg_upload.php>click upload
(200)

# Success! It's a good idea now to just check that this worked by browsing in
# a web browser to your watch list and checking the new items were updated ok (of
# course you can do this in mechshell if you want but I'll leave that out here!).

# Finally, the really cool bit. The mechshell 'script' command will dump out
# the perl code required to perform all of the above actions again if you copy
# them into a perl script:
http://www.freshports.org/pkg_upload.php>script
#!perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;
-snip-

# Also, if you provide a filename as an argument to the 'script' command,
# mechshell will dump all the script commands to that filename:
http://www.freshports.org/pkg_upload.php>script /tmp/freshports_update.pl

# Finally, use 'quit' to exit the mechshell:
http://www.freshports.org/pkg_upload.php>quit
munk@users /home/munk#

Now all that remains is to open up /tmp/freshports_update.pl and tidy the script up so that it's more suitable for automated use via cron. For example, any 'dump' and 'content' commands can be taken out - these would only cause problems anyway if run from a non-interactive shell as used by cron.

We also need to add some code to have the script dump the contents of 'pkg_info -qoa' to a temporary file prior to uploading.

The completed 'quick and dirty' hack looks like this then:

CODE:

#!/usr/bin/perl -w
use strict;
use WWW::Mechanize;
use WWW::Mechanize::FormFiller;
use URI::URL;

# FreshPorts username/pass:
my $user="munk";
my $pass="xxxxx";

# Temp location to store output from 'pkg_info -qoa':
my $mypkg_info="/tmp/freshports/mypkg_info.txt";

# prepare file containing output from: pkg_info -qoa
`pkg_info -qoa > $mypkg_info`;

# Prepare WWW::Mechanize:
my $agent = WWW::Mechanize->new( autocheck => 1 );
my $formfiller = WWW::Mechanize::FormFiller->new();
$agent->env_proxy();

# Login to FreshPorts:
$agent->get('http://www.freshports.org/login.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('UserID', $user); };
{ local $^W; $agent->current_form->value('Password', $pass); };
$agent->submit();

# Submit pkg_info details to FreshPorts pkg_upload page:
$agent->get('http://www.freshports.org/pkg_upload.php');
$agent->form_number(1) if $agent->forms and scalar @{$agent->forms};
{ local $^W; $agent->current_form->value('pkg_info', $mypkg_info); };
$agent->click('upload');

# Remove temporary file:
`rm $mypkg_info`;

After saving the script and making the file executable, an entry can then be added to cron to have the script auto update the list of ports at freshports once a week - or however often you require it to be updated, once a week is more than enough for me. Sorted! :)

Let root see all files with locate

nospam@example.com (munk) — Sat, 18 Nov 2006 10:38:40 +0000

The locate utility on linux was one of the first tools I hit when I made the move to FreeBSD a few years back - knowing where files are is half the battle when you're trying to configure things and find documentation on how to do it. The trouble with locate though as jdarnold mentions in his article 'Locate This!' is that if you build the locate database as 'root', you end up exposing everything to any user that runs the locate command. The other problem he mentions is the locate db is only updated weekly on FreeBSD by default via the periodic system which isn't really enough if you use your system regularly.

I remember thinking along the same lines a while back and after reading through the man pages the solution I found was to create two separate databases - one for root and one for regular users. The 'regular' db is updated on a weekly basis as per the default on FreeBSD via periodic, whereas the other 'root' locate db is built daily in a crontab so I can get the latest up to date details on which files are where.

To get the root db built first you need to create a crontab entry - i put this in /etc/crontab:

CODE:

39 2 * * * root env -i LOCATE_CONFIG=/root/locate/conf/locate.rc /usr/libexec/locate.updatedb > /dev/null 2>&1

This tells the locate.updatedb script to use a separate configuration file - /root/locate/conf/locate.rc - for building root's locate db. The content of /root/locate/conf/locate.rc look like this:

CODE:

FCODES="/root/locate/db/locate.database.root"

which indicates that this db should be built in /root/locate/db/locate.database.root instead of the default locate in /var/db/locate.database. You can safely run the command as root on the commandline to initialize your new db:

CODE:

root@users /root# env -i LOCATE_CONFIG=/root/locate/conf/locate.rc /usr/libexec/locate.updatedb

Once the database is built you can move on to test the new db works ok:

CODE:

root@users /root# locate -d /root/locate/db/locate.database.root .cshrc.root
/root/.cshrc.root

This file is only readable by root, so it seems to work ok.

To make things easier, add a shell alias in root's .cshrc file aliasing 'locate' to the command 'locate -d /root/locate/db/locate.database.root':

CODE:

root@users /root# grep locate $cshrc
alias locate locate -d /root/locate/db/locate.database.root

With the "-d /root/locate/db/locate.database.root" switch, locate will use the db at /root/locate/db/locate.database.root instead of the default /var/db/locate.database and root will be able to use locate to find any files in the filesystem, not just those that are world readable.

Finally, one way to update the regular locate db as root but without making it list every world readable file is to perform the following:

CODE:

#!/bin/sh
# make sure db file exists:
touch /var/db/locate.database

# then change ownership to the nobody user:
chown nobody /var/db/locate.database

# make it writeable by nobody and readable by everyone else:
chmod 644 /var/db/locate.database

# then move on to update the db...
# first make sure we're in the / folder where the db update starts:
cd /

# then finally run the updatedb command as the 'nobody' user:
echo "/usr/libexec/locate.updatedb" | su - -fm nobody

This is basically what the 310.locate periodic script does and results in a locate db that contains only files that are readable by the 'nobody' user - essentially all 'world readable' files.

Comparing the sizes of the root db against the nobody db:

CODE:

root@users /# ls -al /var/db/locate.database /root/locate/db/locate.database.root
-rw-r--r-- 1 root wheel 4070484 Nov 18 02:45 /root/locate/db/locate.database.root
-rw-r--r-- 1 nobody wheel 3280409 Nov 18 11:41 /var/db/locate.database

You can see the size difference there, not as many entries in nobody's db as root's. Just to double check:

CODE:

root@users /root# locate .cshrc.root
/root/bin/ktrace.out
/root/ktrace.out
/usr/local/etc/snort/ktrace.out
root@users /root# echo "locate ktrace.out" | su - -fm nobody
/usr/local/etc/snort/ktrace.out

So from that you can see that 'nobody' can see the ktrace.out files located in /root - apart from root of course :) Sorted.

Expand shell globs using 'ctrl-z'

nospam@example.com (munk) — Sun, 05 Nov 2006 11:22:58 +0000

Just noticed a semi useful feature of the CSH shell (shells in general? not tested it) whilst running 'rm -rf *' in a directory. Got a bit paranoid I was doing something silly (running the command as root), so hit 'ctrl-z' to put the process into the background and the '*' part was expanded in the job control list:

CODE:

[11:21:24] root@users /usr/local/www/web/torrentflux.munk.me.uk# rm -rf *
^Z
[1] + Suspended rm -rf TF_BitTornado adodb downloads images language mods searchEngines themes

Yay. Useful tip #341 you'll probably never use but at some point in the future think mmm... now where did I read about that thing about this thing...

CSH Tips: Auto 'whereis' on the tcsh command line

nospam@example.com (munk) — Wed, 18 Oct 2006 00:10:26 +0000

Just read this entry about the 'whereis' command on Unix and it reminded me of another great shell tip for tcsh users (csh/tcsh on FreeBSD since they're the same thing!) - the shell can 'normalize' any command on the command line if you bind the normalize function to a keystroke - this allows you to easily see how a command would expand after it's executed. Somewhat esoteric without an example, but I use it so much I thought I'd post about it.

First off set a key binding for the normalize command - in the shell type in:

CODE:

bindkey "^W" normalize-command

or add it to ~/.cshrc to make it permanent. Obviously you can set it to whatever keybinding you want, I use ctrl-w.

Now type in any command that you'd use on the commandline and then whilst the cursor is at the end of the command, hit the keystroke you entered for the normalize command - in our case above, ctrl-w.

The command you entered should automatically get expanded to the absolute path of the command. For example if I type in:

CODE:

and then hit 'ctrl-w' whilst the cursor is just after the 's', the result will look like:

CODE:

/bin/ls

Magic!

Like I say I use this function quite a lot on the commandline, particularly when I want to see how a command I enter will get expanded - the normalize function works on aliases as well as just plain commands, so it's quicker to type 'portupgrade^w' than to type in 'alias portupgrade' to see how I've got my portupgrade alias set up. It's also great for quickly editing system executables from the commandline without having to remember where the file/script is or cut/paste the results from 'whereis' or 'locate' etc.

Shell Tip: Review your most commonly used command lines

nospam@example.com (munk) — Tue, 10 Oct 2006 00:22:47 +0000

Just got through reading an interesting article on how to review your most commonly used Unix commands. The idea is to sort the most commonly used commands numerically with a view to maybe shortening the most command ones using aliases, similar in a way to the time saving article mentioned here a while ago.

(The lifehacker article is actually just picking up on the original article on IBM's site entitled Unix productivity tips and is a good read for anyone wanting to improve their efficiency on the shell command line.)

Note for tcsh on FreeBSD, the command you probably want to use is this:

CODE:

history | tail -1000 | awk '{print $3}' | sort | uniq -c | sort -r

which outputs this kind of list for my shell history (listing top 10 used commands using '| head -10'):

CODE:

; history | tail -1000 | awk '{print $3}' | sort | uniq -c | sort -r | head -10
212 sc
158 m
  96 fg
  75 s
  68 cd
  56 vi
  37 ls
  26 grep
  19 man
  18 alias

I'm pretty happy with that, most of the commands are either 1 or 2 character aliases at least :)

Executing webserver commands using su on FreeBSD

nospam@example.com (munk) — Sat, 07 Oct 2006 20:04:03 +0000

On FreeBSD the webserver user - www - doesn't have a shell enabled, so testing any PHP/CGI commands that are executed using a 'system' call can't easily be tested on the commandline.

For example say you wanted to test the result of this PHP call in a shell:

CODE:

exec('echo "hello world"');

If the web user www had a shell - like it does on many linux distros (a bad thing (tm), compromise is easier if you can spawn a shell under the user the webserver runs as) - you could just do:

CODE:

su www
echo "hello world"

and see the results fine. Since www has no shell though on FreeBSD, you can't just 'su' to www.

Instead you can do the following:

CODE:

echo 'echo "hello world"' | su - -f -m www

This runs the command as the 'www' user, but doesn't try to invoke the 'www' user's .cshrc file or modify the environment (which is something to be aware of since it's your shell environment you're using, probably root, make sure there's nothing there that could affect the outcome of the result).

This entry is a memo to myself really, I always forget the 'su - -fm user' thing! Maybe there's a reason not to use this as well, feel free to comment.

Shell Tips: Generate random numbers on the shell commandline

nospam@example.com (munk) — Sat, 07 Oct 2006 19:15:00 +0000

Whilst reading through the shell script for the bsdstats utility I noticed a nice way of generating a random number on the command line:

CODE:

jot -r 1 0 900

Which generates a random integer between 0 and 900.

Useful to remember, have resorted to some wierd and wonderful ways of generating randomness before now (usually involving consumption of alcohol:o).

Improve Your Shell Life With Screen

nospam@example.com (munk) — Mon, 18 Sep 2006 22:20:00 +0000

Screen is one of those utilities you use for a short while and then find you can't live without it. Imagine having a single Putty session running on your client machine, but having the same functionality of hitting <ALT>-F1, <ALT>-F2, <ALT>-F3 to switch easily between 3 different virtual tty's on the live server. Screen allows you to login to a server once and then open as many extra shells as you want all within the same screen session and switch easily between them with simple keystrokes.

A sample session goes something like follows (try this yourself if you're new to screen, remember to install it first!):

Login to the server.
Fire up screen for the first time:

CODE:
[22:43:29] munk@users /home/munk# screen
<screen starts up>
Screen version 4.00.02 (FAU) 5-Dec-03

Copyright (c) 1993-2002 Juergen Weigert, Michael Schroeder
Copyright (c) 1987 Oliver Laumann
...
[Press Space or Return to end.]
Hit space to continue, now you're inside screen. Try running some command there like 'ls -al'.
Now start another shell/window in screen by hitting the keystroke 'ctrl-a c' (c for create). Another screen shell should open up. Try running something in there like 'echo "hello world'.
Now go back to the original screen by hitting the keystroke 'ctrl-a ctrl-a' - you should see the output from the 'ls -al' command still there. To move back to the second screen shell, hit the keystroke again 'ctrl-a ctrl-a' - you should see the output from the 'echo "hello world"' command.
Now try detaching from the screen session. This is the feature that makes screen rock - you can open lots of shells under screen and then when you want to go and do something else, instead of closing each and every shell down, you just 'detach' from screen.

This effectively leaves the shells running in the background - or whatever it is you've got running in the screen shell - and when you want to come back later you can type a simple command and 'attach' back to the screen session with all the shells/processes etc still running right where you left off. Very cool for staying logged into IRC servers whilst you're away from the machine - even get logged into two different servers in two different shells running in screen.

To detach from a screen session, just type in 'ctrl-a ctrl-d' - you should see the text '[detached]'. You can close down the shell you're left in and go off and do something else now. In fact you don't even need to detach from screen when you want to leave a screen session - say you want to shut your workstation down quickly and you have putty running an instance of screen. Just kill the putty process and job done (I actually just hit the 'power' button on the PC and let windows kill everything for me:)). The screen session will still be ok in the background and when you come back later you can just resume the session fine.
Ok now try and reattach to the screen process we detached from earlier. To do this, login to the server as the user you started screen as and type in 'screen -d -r' - I have this aliased to 'sc' because I use it so much. After entering that command, screen should reattach itself to the session you left off above, with the two screen shells/windows running (try 'ctrl-a ctrl-a' again). Magic :)
To quit one of the shells from inside screen, just issue the command you'd usually use to exit the shell - usually 'exit' or 'logout' or the 'ctrl-d' keystroke depending on what shell you're using. When you exit out of the last shell running under screen, screen itself will exit.

You can find all the keystrokes for getting things done in screen are listed in the man page 'man screen'.

The list of most common keystrokes in screen to help you get around though are:

ctrl-a c - create a new shell / window
ctrl-a ctrl-a - switch to the shell/window previously displayed
ctrl-a n - switch to the next shell/window (useful if you have more than 2 windows/shells running)
ctrl-a p - switch to the previous shell/window (useful if you have more than 2 windows/shells running)
ctrl-a d - exit out of the screen session

Note also that normal shell keystrokes that start with 'ctrl-a' - these have to be 'escaped' before you execute them. The main culprit is the 'ctrl-a' keystroke to get to the start of a line in the shell - to do this from within screen you have to type in 'ctrl-a a' - that's 'hold down ctrl and hit a at the same time, let go, then hit a again(!). Slightly confusing at first but after a short while it becomes second nature.

There are a load of other cool things you can do in screen, like naming windows/screens, splitting windows so you can have 2 shells on the same display (one shell at the top half of the display, one at the bottom), switch to a screen quickly using 'ctrl-a 0...9' for screen number 0 ... 9 etc - the commands above though are the ones I use the most and you can survive easily with just those - let me know if I left anything out :)

Shell Tips: Improved Shell Usage Using Temp Files

nospam@example.com (munk) — Mon, 11 Sep 2006 16:29:51 +0000

Most of the time when working in a shell, it's the output from commands that you're interested in. Using this tip you can make it easier to capture the output from a shell command to a temp file and not worry about cleaning up after you.

The gist of the tip is very simple - create a temp directory inside your home directory - ie /home/munk/tmp - then setup a shell environment variable called $tmp which points to /home/munk/tmp/tmp.$$ - where $$ represents the process id of the current login shell. Finally add a line to your ~/.logout file that deletes anything within /home/munk/tmp. With this in place you can then easily redirect output from commands straight to that $tmp file, page through it as you want and then not worry about having to remove it manually at the end of the login session.

This may sound convoluted but is something I would find hard working without now. It takes a little while to get used to remembering to use '>$tmp' at the end of commands to capture the output, but once you get used to it it's like second nature and you know you have the output around to go back to during the login session.

This tip can be applied to any shell, but I personally use the CSH Shell so the code snippets below will focus on the syntax used in the CSH rc file, ~/.cshrc and ~/.logout. The same principle applies to other shells though and shouldn't be too hard to set up.

Here are the steps for setting the system up:

Create a temp directory inside your home directory and make sure only your user has access to it:

CODE:
mkdir ~/tmp
chmod 700 ~/tmp

To make the system more portable you can also add the following to ~/.cshrc:

CODE:
[ !-d ~/tmp ] && mkdir ~/tmp && chmod 700 ~/tmp
Edit your ~/.cshrc file and add the following environment variable, $tmp:

CODE:
setenv tmp ~/tmp/tmp.$$

The $tmp filename variable will be unique to this login session, $$ is the special shell variable which contains the process ID of the current process - in this case the PID will be that of the current shell login session. You can see the value of the $tmp variable by using 'echo':

CODE:
echo $tmp
/home/munk/tmp/tmp.26119

Also you can verify the current shell has that PID:

CODE:
ps -auxww 26119
munk 26119 0.0 0.2 1544 1068 p2 Ds 5:39PM 0:00.13 /bin/tcsh
Edit or create ~/.logout to include the following:

CODE:
; cat ~/.logout
rm -rf ~/tmp/ && mkdir ~/tmp && chown munk:munk ~/tmp

You might want something in there to change perms on the directory to 700 as well if you want to protect anything dumped into $tmp.

Now you're ready to use the system. Logout and back in again or issue 'rehash' in CSH to setup the $tmp variable.

Use the $tmp variable anywhere where you have a lot of output scrolling along that you want to be able to page through easily any time during the shell session:

CODE:

> grep mess $messages>$tmp
> head $tmp
Sep  1 00:00:45 users last message repeated 4 times
Sep  3 00:54:04 users last message repeated 188 times
Sep  3 14:53:54 users last message repeated 119 times
...

If you want to use a different 'temp' file during the session, just add a number onto the end of $tmp - ie $tmp.1 etc:

CODE:

; tail $messages > $tmp.2
; cat $tmp.2
Sep 11 09:53:30 users sshd[12583]: login_getclass: unknown class 'root'
Sep 11 09:53:30 users sshd[12584]: warning: /etc/hosts.allow, line 29
Sep 11 09:53:31 users sshd[12585]: login_getclass: unknown class 'root'
...

This can be useful if you need to keep the contents of the first $tmp file and work with the content of another command easily.

Another extra tip to use with this one is to create a shell alias called 'myread'. What the myread alias does is have any output from a command sent both to the current tty console as regular output but at the same time have it also piped to the $tmp file and then finally open the $tmp file in an editor of your choice. Here's the 'myread' CSH alias - syntax for other shells will differ slightly:

CODE:

alias myread 'tee $tmp && vim $tmp'

That way instead of doing this:

CODE:

> grep something somefile > $tmp
> vim $tmp

you can just do this instead:

CODE:

> grep something somefile | myread

CSH Tips: Create Variables Everywhere

nospam@example.com (munk) — Thu, 07 Sep 2006 00:15:34 +0000

Shell environment variables allow you to easily access commonly used filenames/strings without the hassle of typing them out in longhand.

For example instead of typing out:

CODE:

cd /usr/local/etc/rc.d

it's a lot quicker just typing:

CODE:

cd $rc_dir

Even better, command line completion in CSH allows you to use autocomplete on env variables, so you only need to type the first few chars of the variable, hit and have a list of possible completions given:

CODE:

[1:36:07] munk@users /home/munk# cd $r<TAB>
rc_dir rcconf rejectlog rubybase

giving all the env variables that start with 'r'.

Adding env variables can be done on the commandline using:

CODE:

setenv var_name string

where 'var_name' is the name of the variable you want to create and 'string' is the string you want to assign to the vbl.

However the best way is to add 'setenv' lines to your ~/.cshrc file so the variables are ready to use when you login:

CODE:

; grep setenv $cshrc | head
setenv munk_dir "/home/munk/"
        setenv PAGER "less -i"
        setenv EDITOR vim
        setenv PAGER "less -i"
setenv  BLOCKSIZE       K
setenv  MAIL            /var/mail/$USER
                setenv TERM xterm-color
# Common setenvs:
setenv tmp                              /home/munk/tmp/tmp.$$
setenv MONTH                    `date "+%m"`

Some of my fav env vbls:

CODE:

# useful for redirecting output to a temp file unique to this login session - 'grep something somedir -ri > $tmp':
setenv tmp              /home/munk/tmp/tmp.$$

# useful for creating timestamps in filenames - tar zcvf sometar.$DATE.tar.gz somedir:
setenv DATE             `date "+%Ymd"`

# easy access to /var/log/messages - 'tail -f $messages' etc:
setenv messages /var/log/messages

# start/stop/restart system daemons quicker with '$rc_dir/daemon.sh restart':
setenv rc_dir           /usr/local/etc/rc.d

# access the perl base directory quicker:
setenv perlbase /usr/local/lib/perl5/site_perl/5.8.8

# crontab:
setenv crontab /etc/crontab

# quickly edit ~/.cshrc - vi $cshrc:
setenv cshrc        /home/munk/.cshrc

# change the order of sourceforge websites that portupgrade fetches from by default:
setenv MASTER_SITE_SOURCEFORGE \
    "http://eu.dl.sourceforge.net/%SUBDIR%/ \
    http://us.dl.sourceforge.net/%SUBDIR%/ \
    ftp://us.dl.sourceforge.net/pub/sourceforge/%SUBDIR%/ \
    ftp://ftp.kddlabs.co.jp/sourceforge/%SUBDIR%/ \
    ftp://ftp.chg.ru/pub/sourceforge/%SUBDIR%/"

Any time I find myself using a string on the commandline more than half a dozen times in a session I usually add a variable in $cshrc for it - actually using a shell alias 'cshrc_edit', but that's another top tip and needs an article all to itself :)

Solving permission problems with parsepath.pl

nospam@example.com (munk) — Mon, 04 Sep 2006 22:41:00 +0000

parsepath.pl is a brilliant perl script for fixing permissions problems on Unix based platforms by Jeremy Mates. Probably the most common type of permission problem from a sysadmin/webmaster's viewpoint is uploading a file to a directory in a website's document root folder and then trying to access the file or script in a web browser only to get the dreaded 403 error message:

Forbidden
You don't have permission to access /foo/bar/test.php on this server.

Most time the solution is very simple, just change the permissions on 'test.php' to make sure the user the webserver runs as can read the file correctly - the simplest and most common method being to change the mode of the file to '755':

CODE:

chmod 755 test.php

Unfortunately sometimes it's not that easy and many times you see users asking 'I'm getting 'access denied' errors even though I've changed the perms to 755'. The problem is that one of the subdirectories that the 'test.php' file lives in has permissions set so that the webserver can't read the file properly. Now that's where the headache comes in :)

However, parsepath.pl can take the headache out of fixing permissions problems.

Say you have a website document root directory tree /usr/local/www/web/www.munk.me.uk/foo/bar and you upload a web script 'test.php' into that directory. You try and access the file in a webbrowser but get the 403 permission denied error above. First off you check the permissions on the file itself:

CODE:

[23:58:17] root@users /usr/local/www/web/www.munk.me.uk/foo/bar# ; ls -l
total 0
-rwxr-xr-x 1 www www 0 Sep 4 23:39 test.php

That looks ok, with permissions 755 and the owner/group set to 'www' the webserver user 'www' should be able to read the file ok. So in this case the problem must be with the permissions on one of the parent subdirectories. The old method of working out the perms would be either to trawl one by one through each directory checking the perms on each subdirectory or to change the permissions recursively on the document root folder so all subfolders have the read bit set for the webserver user/group.

With parsepath.pl things are a lot simpler though - just run the following command:

CODE:

[0:03:21] root@users /usr/local/www/web/www.munk.me.uk/foo/bar# parsepath.pl user=www +r test.php
! group=www +rx fails: d 0700 root:www /usr/local/www/web/www.munk.me.uk/foo
! unix-other +rx fails: d 0750 root:wheel /usr/local/www/web/www.munk.me.uk/foo/bar

With this command parsepath.pl recurses through each subdirectory below the file/path you feed it on the commandline and tells you the permissions problems - if any - for the user 'www' (the user=www argument) to read (the +r argument) the file 'test.php'.

In the output, we're told that permissions to read the test.php by the user www fails on two counts:

CODE:

# the group bit on the folder 'foo' doesn't have the +rx flag set:
! group=www +rx fails: d 0700 root:www /usr/local/www/web/www.munk.me.uk/foo

# the other bit on the folder 'bar' doesn't have the +rx flag set:
! unix-other +rx fails: d 0750 root:wheel /usr/local/www/web/www.munk.me.uk/foo/bar

With this information it's easy enough to go in and make the changes necessary to fix the problem using 'chmod g+rx foo foo/bar'.

There are other ways of invoking parsepath.pl though. Running it just with a file/path as an argument it'll tell you the permissions on each subdirectory under it:

CODE:

[0:10:33] root@users /usr/local/www/web/www.munk.me.uk/foo/bar#
> parsepath.pl /usr/local/www/web/www.munk.me.uk/foo/bar/test.php
% /usr/local/www/web/www.munk.me.uk/foo/bar/test.php
d 0755 root:wheel /
d 0755 root:wheel /usr
d 0755 root:wheel /usr/local
d 0755 root:wheel /usr/local/www
d 0770 www:wheel /usr/local/www/web
d 0750 www:www /usr/local/www/web/www.munk.me.uk
d 0700 root:www /usr/local/www/web/www.munk.me.uk/foo
d 0750 root:wheel /usr/local/www/web/www.munk.me.uk/foo/bar
f 0755 root:www /usr/local/www/web/www.munk.me.uk/foo/bar/test.php

which can is better to see a whole tree in one go.

No permissions were harmed in the making of this article!

I'll include the parsepath.pl script in the extended article just in case the original ever gets lost - big credit of course goes to the author of the script, Jeremy Mates. His site is actually very interesting from a sysadmin's point of view containing lots of interesting admin scripts and thoughts on system administration in general - spent quite a while grazing through his stuff there - cheers Jeremy.

Continue reading "Solving permission problems with parsepath.pl"