Wed, 09 Jul 2008
Amazon^WLoveFilm DVD Rentals
I've been a happy Amazon UK DVD rental customer for the last couple of
years. They've got a wide selection, the DVDs ship fast, come in
separate envelopes and in nice sturdy plastic cases. In nearly 200 DVDs
I've had three that were unplayable and only one that got lost in
transit - a replacement for which was sent the same day.
'Luckily' for me Amazons DVD rentals are now handled by LoveFilm. I had my first batch through last week and I'm less than impressed. All three films came in a single envelope (so I have to watch them all before I can sent them back for replacement), they are in flimsy paper containers and out of my first two one is broken - with a split straight through. This is not the best start I could have hoped for.
There's a reason Amazon gets a lot of my online cash - they provide a damn good service.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/07/09 19:52 | /tools/online | Permanent link to this entry | This entry + same date
Tue, 08 Jul 2008
Dear Lazyweb - Command Line YSlow!
The title pretty much says it all, I'd like a command line version of YSlow! (what is it with
Yahoo and !s) that I can run from cron and import in to a nice
spreadsheet for trending and site comparisons.
I don't have XUL on my list of things to play with so I'll give it a couple of months and watch someone else implement it. Hopefully.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/07/08 20:59 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 02 Jan 2008
YSlow - a Grade A Tool
I spent a couple of hours running the YSlow
FireFox extension against the main website for one
of my little side projects and I couldn't stop fiddling with the sites
config until I got the score up. Improving a category until you get an
'A' gives you that same moment of satisfaction as all your tests passing
or a file restore working perfectly.
Due to not being amazingly wealthy I cheated with the content delivery network stage and just overrode it with the sites own name.
I'd happily recommend this tool to anyone who wants to learn some good habits. With the exception of the CDN rule it's hard to argue with most of its suggestions. The performance grade is a nice way of giving near instant feedback and encouraging at least basic sensible behaviour.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/01/02 22:52 | /tools/firefox | Permanent link to this entry | This entry + same date
Mon, 25 Jun 2007
Navigating Commented Config Files
The current trend with config files is to fill them with comments (let's
ignore the fact this isn't a substitute for documentation) and while this
is helpful watching people arrow through them line by line looking for
active options drives me nuts.
If you're using vim (as all good people
do ;)) you can jump from uncommented directive to uncommented directive
with /^[^#] as a search. Pressing n will then
move you to the next uncommented option. And save me from pulling out those
precious few hairs I have left.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/25 22:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 03 Jun 2007
Nagios - Simple Trender
Continuing the release of my Nagios code - here's my Nagios
Simple Trender. It parses Nagios logs and builds a horizontal barchart
for host outages, service warnings and criticals. It's nothing fancy (and
the results are a little unpretty) but it does make the attention seeking
services and hosts very easy to find.
While the tool isn't that technically complex I've found it useful in justifying my time on certain parts of the infrastructure. Being able to show how bad NTP is for example (we had 216 NTP sync problems last month, this month we had 36; and most of those are one machine with a bad clock) on a very simple chart makes it easier to get buy in from above. And next month you can show them how much of a positive impact the work had.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 11:29 | /tools/commandline | Permanent link to this entry | This entry + same date
The Nagios Tag Cloud
We use the Nagios monitoring system
at work (in fact we use four installs of it for physically isolated
networks) and while it's damn useful (and service checks are easy to
create or extend) it's a little lacking in higher level trending and
visualisation tools. Well, at least the very old version we run suffers
from this.
Thankfully I work for a company that invests time in its core tools. Over the last couple of hackdays I've written two small scripts for parsing Nagios logfiles and presenting the information in a different, slightly more grouped way. The first of these is the Nagios TagCloud - which has a very descriptive name :)
When invoked (I typically use nagiosclouds.pl /log/files/*.log >
/webdir/nagios_tagcloud.html from a cronjob) it'll run through the log
files and produce a HTML page containing 3 tag clouds, one for host
outages, one for service warnings and one for service criticals. Tag clouds
don't suit everyones work style but I came away from running ours with a
couple of action points so I think they're useful enough to glance at once
a month.
I should note the perl module that generates the tag cloud is Leon Brocards HTML::TagCloud and the CSS was graciously given to me by Alex Monney after he burned his eyes looking at my first version.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 11:08 | /tools/commandline | Permanent link to this entry | This entry + same date
The Perk of $HOME - .bash_logout
It's not a well kept secret but I'm still surprised by how many people have
never encountered .bash_logout. Its purpose is pretty simple,
if you use the BASH shell it'll be executed when you log out (see, a well
named file!)
So what's it for? Well, I use mine to invalidate any sudo
sessions I've got open (sudo -k), clear the
screen -in case it's a local session - and nuke a history file or two.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 10:47 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 06 May 2007
df Output Ordering
Sometimes questions come up that you know you should know the answer to but
you just don't. My recent one was "how does df choose the
output order?" The man page doesn't mention the logic behind it and a quick
strace shows it pulls its data from /proc/mounts
(which you'd expect) and returns the output in the same order. So logically
the question becomes how does /proc/mounts order things?
It's not exactly an important question but I can see how this ends - and it involves source code.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/05/06 12:46 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 27 Mar 2007
VMWare Free Converter - First Thoughts
While we're a Xen shop I've always been a VMWare fan and I
had the chance to take a look at the free (as in beer) VMWare Converter
Starter today. We've got a couple of old Windows machines with no
installation documents or run books so when working towards making them
reproducible grabbing a whole system image is a great first step.
The first machine I tried it on has a very unhappy hard drive (yes, it's my work laptop) and the converter refused to play past 5% of the disk; me thinks it's time to verify my backups. The second machine was a Windows 2000 server (amusingly running VMWare server). The converter required a reboot (which it didn't on the laptop running Windows XP) after installation but made an image afterwards without any complaints and with the machine up and running.
I've not had the time to fully dig in to how well this'll work on the more awkward machines (boxes with more than 2 CPUs, apps that expect hardware access, VMWare tools not installed etc.) but the image of my trial machine (which was written out to a UNC path) came up quite quickly and all the settings I checked were correct.
I like the tool, it provides a nice revertable image for me to dissect so I can work out what's on the machines with out being a resource drain on the live servers. It's simple to use, has a nice GUI, a great price tag and will make a painful task a lot simpler. In a worst case scenario the images can also be pushed in to service as a stop gap in order to reduce the MTTR of the original servers. Oh, you can also use it to help bootstrap server consolidation, but that'll never take off... ;)
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/03/27 23:48 | /tools/gui | Permanent link to this entry | This entry + same date
Sat, 10 Mar 2007
FRDNS Revisions - now with added ping checks!
I originally wrote frdns to
find and warn about inconsistencies in forward and reverse DNS records.
At the time I was also using a tool called hawk to show both IPs that
didn't have a reverse record and reverse records that didn't have a
responding IP address associated with them (we had a lot of orphaned
records).
While hawk did the job it required a MySQL instance, a daemon process
and an apache server to function - which was a PITA when it had to be moved
to another server. So I improvised. The first step was adding a
-p option to frdns that makes the program ping each IP
specified and flag the address if it doesn't have a reverse record. This
points out IPs that don't have DNS records. As for the no longer needed
records I've got a different tool for that - but that's for another blog
post.
I've also made frdns log both run time and how many issues it flags to syslog. The ping check can take a while so I added this to help me keep an eye on its performance. I did think about using one of the asynchronous DNS libraries to improve performance but we're only running it once a day to pick up mistakes so a long runtime isn't a huge issue.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/03/10 22:46 | /tools/commandline | Permanent link to this entry | This entry + same date
Fri, 09 Mar 2007
ls and the Missing Argument
When it comes to command line options GNU ls already uses
most of the alphabet, so for my own sanity can someone implement a
-j that doesn't change the behaviour much from a ls
-alh? It's my most common typo and I'm willing to offer beer to
remove the problem.
I could learn to type better but this is easier ;)
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/03/09 08:34 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 01 Feb 2007
Ping The Host Table - UGU Tip
I'm not too keen on yesterdays
UGU tip of the day
and it doesn't take much to make it work a chunk better, so I thought I'd
whine about it on my blog.
Here's the original snippet:
grep -v "#" /etc/hosts | awk '{print $1}' | while read host
do
ping -c 1 $host
done
But this has some very fixable caveats. It doesn't deal with blank lines, it'll try and ping IPv6 addresses (and too many distros put IPv6 entries in the host table these days - even if you disable the IPv6 options) and it will ignore any lines that have a comment, even if the comment is after the field we want. So I wrote my own version (which I can't see me ever using)
for host in `awk '! /^#|^$|::/ { print $1 }' /etc/hosts`
do
ping -c 1 $host
done
Mine does deal with blank lines (^$), only drops a comment if it's at the start of the line (^#) and skips all IPv6 addresses (::). If you want to golf it down some more you can even kill both the loop and iterator variable and use xargs instead. But I'm not that bored.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/02/01 07:01 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 07 Jan 2007
The del.icio.us de.dup.er
I like del.icio.us and I've been using it
for a long while now, but what used to be one of the more handy features,
the ability to subscribe to a tag, like 'ruby' or 'linux', has gradually
become less useful as more and more people find old links or repost the
same link. Again. And again. And, well, you get the idea.
So I wrote the del.icio.us de.dup.er script, a small perl cgi that sits between you and del.icio.us and weeds out any duplicate links. I don't know how useful it'll be for other people but I installed it and when comparing the amount of posts it returns to those in the unfiltered tag I'm already seeing a lot less traffic. This is only the first draft (it needs a little love and a chunk of re-writing) but it works. So I thought I'd post it. To run it you'll need a webserver capable of running perl cgi script, a couple of non-core perl modules and an area on disk where it can write its state; it maintains a single state file for each tag. I considered making it run as a hosted service to remove these preqs but that was more than I need right now.
Notes: Anyone who hits the cgi can force it to update and potentially stop you seeing certain links, I get around this by putting in in a secure (HTTP Auth protected) part of my site. It's also got a timeout built in, a defined number of days after it first logs a site (30 days by default) it'll let it through again. And store it for another 30 days.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/01/07 20:40 | /tools/online | Permanent link to this entry | This entry + same date
Thu, 04 Jan 2007
Extending PkgWatcher to work with Other Operating Systems
So now I've Announced
PkgWatcher people are actually starting to use it, the optimistic curs!
The first question's already come in and it's one I can actually answer:
how do you extend it to work on other operating systems?
It's actually pretty easy, first you need to make an addition in
installed_packages. This function works out which OS you're
running on and returns the respective subroutine that understands your
package manager. Which brings us on to step 2.
You now need to add that sub, the two existing examples are pretty simple and show two possible approaches. The goal is to have the function return an array of all the packages installed; how you get this is up to you. But please don't shell out and use awk, pkgwatcher is a perl program after all. And that's it, two small bits of code and you should be up and running.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/01/04 20:44 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 02 Jan 2007
PkgWatcher - Initial Release
When it comes to servers, some packages should be everywhere, some should
be
banned and there are always the edge cases - be it a build host that
requires GCC or a webserver that needs a full complement of packaged perl
modules. While a decent system imaging or ad-hoc change system will help
keep the discrepancies down nothing beats a system level check that
verifies your assumptions. And PgkWatcher is that
check.
The script's easy to use, copy it to the machine, install
Parse::Debian::Packages (via CPAN or apt-get) if you're on a
Debian system and then populate any (or all of) a required packages,
prohibited packages and local packages list. And then run the command with
the files, each line of which should contain a single package name, passed
as options (pkgwatcher -h will show you some examples) and see
what it outputs. The script will tell you about anything that should be
installed and isn't, shouldn't be installed and is, and any packages that
are listed in both files. Which I consider a configuration error. And now to
an added feature, the local package list. This is the last list checked and
allows you to do host specific overrides. If, for example, you banned GCC
on all your servers you'd add it to the prohibited package list. And if you
had a build host that required GCC you'd have to either leave the check off
or customise the file for that one machine. By adding GCC to a local
packages file you can override that one package on that single host.
PkgWatcher was designed to run under Nagios but works just as well as an ad-hoc command line tool. Although without centralised management, keeping the required and prohibited lists up-to-date and in sync could become a hassle. And now some notes, it understands the RPM and DPKG packaging systems (and it's pretty easy to add additional ones), it's written in pure-perl (so it's easy to move around) and it's quite forgiving. If a package isn't on its required or prohibited lists then it does nothing about it. This is both because I'm pragmatic (a good deployment strategy is a better solution to keeping hundreds of machines in check) and because the environment I've written it for has a lot of legacy systems. And being overly strict means you never gain any ground.
What's next for it? It'll soon be plugged in to the configuration templating system we use to get automatic package checking based upon the services defined for that type of host (with package list generation based on the declared OS type).
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/01/02 22:42 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 17 Oct 2006
Check Disk... Checker?
You start off with a couple of partitions. You add a MySQL instance and
put it on a new logical volume. You break its logging out to a different
volume group for performance reasons. You take a snapshot for query
tuning and mount that. You add a chunk of disk for a
short experiment you were going to try... thanks to legacy,
laziness and easy to use LUNs you eventually end up with more mount
points than you know what to do with. And at the worst possible moment
one of them will fill and you'll discover you forgot to add it to Nagios for monitoring. Or you inherit a bundle
of crack fueled servers that have been "evolved" and never gifted with
decent monitoring.
The check_disk_checker.pl script was written to help find mount points that you're not monitoring. It scans through your local Nagios NRPE config files, looks at your current mount points, and complains about any mounted partitions that are not being checked according to the local NRPE configuration files. Of course there is nothing to say that what you have locally is what the remote Nagios is polling but that's outside the scope of this post.
check_disk_checker.pl shells out and grabs a list of all mounted partitions. It then
pulls a list of check-disk lines out of any config files
matching nrpe*.conf or nrpe*.cfg (our local naming scheme) that live
in /etc/nagios. It then extracts the partitions each one checks (it
grabs the value following a -p argument) and complains if it
doesn't find a check for each mounted partition. The script can be run
under Nagios as a plugin or stand-alone for help controlling a legacy
system.
It also plays well with the Linux Check Mounted Disks Nagios Plugin...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/10/17 00:04 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 01 Oct 2006
Linux Check Mounted Disks Nagios Plugin
For my own use as much as anyone elses... One of the problems that's
haunted me at least once per company I've worked at as a tech is "the
disappearing partition". It's there, it's accessible, and it should be
persistent across boots. But it isn't! The machine reboots and
then you discover that the database partition is no longer visible.
The check mounted
disks Nagios plugin looks at the mounted partitions and compares them
to what's in /etc/fstab (minus a couple of things like cd
drives, floppy disks, swap partitions etc). And warns if there are any
discrepancies that'll bite you on a reboot. It also round trips and makes
sure what /etc/fstab thinks is mounted is actually there.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/10/01 23:01 | /tools/commandline | Permanent link to this entry | This entry + same date
Fri, 29 Sep 2006
ps Problems
ps is an incredibly flexible command but it also has a
checkered maintenance history in the Linux world. Yesterday I needed to
output just the username, the command and any arguments passed to it. And
it was hell. After reading through the man page a couple of times I settled
on the following: ps -e -o user,args. But this doesn't
work.
It shows the command and the full arguments but it trunks the username
at 8 characters (which doesn't help with things like exim on Debian - which
has a username of Debian-exim). I then tried switching the order around to
see what happens, and was surprised when ps decided to
truncate the command and argument details at a seemingly arbitrary (but
consistant) point. GAH!
In the end I was pointed at a more correct, but ugly and not obvious
from the man page, answer; supply a width to the arguments. So to show
multiple fields in ps and not have one of them truncated you need a command
like this: ps -e -o user:20,args. And a smart friend like
Paul.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/29 08:03 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 03 Sep 2006
Open Ports Nagios Check
A machine should run a defined set of ports, if any of them are not
listening you've got a problem. If any others are open then you've
potentially got an even bigger problem. The Check Open
Ports Nagios Check accepts a list of IPv4 TCP and UDP ports and
reports if any of the expected ones go away or any others are detected
as listening.
This also partially scratches one of my own itches, I've had a couple of daemons (MySQL in particular) start after a package upgrade without my knowing it. With this script and a little cron it won't happen again. It's probably worth mentioning that while this script is built to run within Nagios it will work stand-alone.
Note: this script is more for detecting misconfigurations than for
security. Most root kits mask the ports they've opened so they won't appear
through netstat, which this command uses.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/03 12:56 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 02 Sep 2006
Bugzilla Tag Blosxom plugin
One of the small developer blogs I host has a number of people linking
to, and complaining about, the bugs present in different Free Software
projects. After watching one of them open a text file, dig through the
links, pull out the wrong one and eventually get the right URL I decided to write a
small Blosxom plugin to make the process easier.
The Bugzilla Tag Blosxom plugin lets you define shortcuts to a number of Bugzilla servers, and a default one, which you can then link to using the following syntax in your blosxom posts:
<bug "redhat">117894</bug>
<bug "mozilla">84752</bug>
And the links look like this:
Red Hat bug ID 117894
Mozilla.org bug ID 84752
For full details have a look at the Bugzilla Tag Blosxom plugin source.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/02 16:22 | /tools/online | Permanent link to this entry | This entry + same date
Copyright © 2000-2005 Dean Wilson
