Tue, 08 Jul 2008
Dear Lazyweb - Command Line YSlow!
The title pretty much says it all, I'd like a command line version of YSlow! (what is it with
Yahoo and !s) that I can run from cron and import in to a nice
spreadsheet for trending and site comparisons.
I don't have XUL on my list of things to play with so I'll give it a couple of months and watch someone else implement it. Hopefully.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/07/08 20:59 | /tools/commandline | Permanent link to this entry | This entry + same date
Mon, 25 Jun 2007
Navigating Commented Config Files
The current trend with config files is to fill them with comments (let's
ignore the fact this isn't a substitute for documentation) and while this
is helpful watching people arrow through them line by line looking for
active options drives me nuts.
If you're using vim (as all good people
do ;)) you can jump from uncommented directive to uncommented directive
with /^[^#] as a search. Pressing n will then
move you to the next uncommented option. And save me from pulling out those
precious few hairs I have left.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/25 22:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 03 Jun 2007
Nagios - Simple Trender
Continuing the release of my Nagios code - here's my Nagios
Simple Trender. It parses Nagios logs and builds a horizontal barchart
for host outages, service warnings and criticals. It's nothing fancy (and
the results are a little unpretty) but it does make the attention seeking
services and hosts very easy to find.
While the tool isn't that technically complex I've found it useful in justifying my time on certain parts of the infrastructure. Being able to show how bad NTP is for example (we had 216 NTP sync problems last month, this month we had 36; and most of those are one machine with a bad clock) on a very simple chart makes it easier to get buy in from above. And next month you can show them how much of a positive impact the work had.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 11:29 | /tools/commandline | Permanent link to this entry | This entry + same date
The Nagios Tag Cloud
We use the Nagios monitoring system
at work (in fact we use four installs of it for physically isolated
networks) and while it's damn useful (and service checks are easy to
create or extend) it's a little lacking in higher level trending and
visualisation tools. Well, at least the very old version we run suffers
from this.
Thankfully I work for a company that invests time in its core tools. Over the last couple of hackdays I've written two small scripts for parsing Nagios logfiles and presenting the information in a different, slightly more grouped way. The first of these is the Nagios TagCloud - which has a very descriptive name :)
When invoked (I typically use nagiosclouds.pl /log/files/*.log >
/webdir/nagios_tagcloud.html from a cronjob) it'll run through the log
files and produce a HTML page containing 3 tag clouds, one for host
outages, one for service warnings and one for service criticals. Tag clouds
don't suit everyones work style but I came away from running ours with a
couple of action points so I think they're useful enough to glance at once
a month.
I should note the perl module that generates the tag cloud is Leon Brocards HTML::TagCloud and the CSS was graciously given to me by Alex Monney after he burned his eyes looking at my first version.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 11:08 | /tools/commandline | Permanent link to this entry | This entry + same date
The Perk of $HOME - .bash_logout
It's not a well kept secret but I'm still surprised by how many people have
never encountered .bash_logout. Its purpose is pretty simple,
if you use the BASH shell it'll be executed when you log out (see, a well
named file!)
So what's it for? Well, I use mine to invalidate any sudo
sessions I've got open (sudo -k), clear the
screen -in case it's a local session - and nuke a history file or two.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 10:47 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 06 May 2007
df Output Ordering
Sometimes questions come up that you know you should know the answer to but
you just don't. My recent one was "how does df choose the
output order?" The man page doesn't mention the logic behind it and a quick
strace shows it pulls its data from /proc/mounts
(which you'd expect) and returns the output in the same order. So logically
the question becomes how does /proc/mounts order things?
It's not exactly an important question but I can see how this ends - and it involves source code.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/05/06 12:46 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 10 Mar 2007
FRDNS Revisions - now with added ping checks!
I originally wrote frdns to
find and warn about inconsistencies in forward and reverse DNS records.
At the time I was also using a tool called hawk to show both IPs that
didn't have a reverse record and reverse records that didn't have a
responding IP address associated with them (we had a lot of orphaned
records).
While hawk did the job it required a MySQL instance, a daemon process
and an apache server to function - which was a PITA when it had to be moved
to another server. So I improvised. The first step was adding a
-p option to frdns that makes the program ping each IP
specified and flag the address if it doesn't have a reverse record. This
points out IPs that don't have DNS records. As for the no longer needed
records I've got a different tool for that - but that's for another blog
post.
I've also made frdns log both run time and how many issues it flags to syslog. The ping check can take a while so I added this to help me keep an eye on its performance. I did think about using one of the asynchronous DNS libraries to improve performance but we're only running it once a day to pick up mistakes so a long runtime isn't a huge issue.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/03/10 22:46 | /tools/commandline | Permanent link to this entry | This entry + same date
Fri, 09 Mar 2007
ls and the Missing Argument
When it comes to command line options GNU ls already uses
most of the alphabet, so for my own sanity can someone implement a
-j that doesn't change the behaviour much from a ls
-alh? It's my most common typo and I'm willing to offer beer to
remove the problem.
I could learn to type better but this is easier ;)
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/03/09 08:34 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 01 Feb 2007
Ping The Host Table - UGU Tip
I'm not too keen on yesterdays
UGU tip of the day
and it doesn't take much to make it work a chunk better, so I thought I'd
whine about it on my blog.
Here's the original snippet:
grep -v "#" /etc/hosts | awk '{print $1}' | while read host
do
ping -c 1 $host
done
But this has some very fixable caveats. It doesn't deal with blank lines, it'll try and ping IPv6 addresses (and too many distros put IPv6 entries in the host table these days - even if you disable the IPv6 options) and it will ignore any lines that have a comment, even if the comment is after the field we want. So I wrote my own version (which I can't see me ever using)
for host in `awk '! /^#|^$|::/ { print $1 }' /etc/hosts`
do
ping -c 1 $host
done
Mine does deal with blank lines (^$), only drops a comment if it's at the start of the line (^#) and skips all IPv6 addresses (::). If you want to golf it down some more you can even kill both the loop and iterator variable and use xargs instead. But I'm not that bored.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/02/01 07:01 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 04 Jan 2007
Extending PkgWatcher to work with Other Operating Systems
So now I've Announced
PkgWatcher people are actually starting to use it, the optimistic curs!
The first question's already come in and it's one I can actually answer:
how do you extend it to work on other operating systems?
It's actually pretty easy, first you need to make an addition in
installed_packages. This function works out which OS you're
running on and returns the respective subroutine that understands your
package manager. Which brings us on to step 2.
You now need to add that sub, the two existing examples are pretty simple and show two possible approaches. The goal is to have the function return an array of all the packages installed; how you get this is up to you. But please don't shell out and use awk, pkgwatcher is a perl program after all. And that's it, two small bits of code and you should be up and running.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/01/04 20:44 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 02 Jan 2007
PkgWatcher - Initial Release
When it comes to servers, some packages should be everywhere, some should
be
banned and there are always the edge cases - be it a build host that
requires GCC or a webserver that needs a full complement of packaged perl
modules. While a decent system imaging or ad-hoc change system will help
keep the discrepancies down nothing beats a system level check that
verifies your assumptions. And PgkWatcher is that
check.
The script's easy to use, copy it to the machine, install
Parse::Debian::Packages (via CPAN or apt-get) if you're on a
Debian system and then populate any (or all of) a required packages,
prohibited packages and local packages list. And then run the command with
the files, each line of which should contain a single package name, passed
as options (pkgwatcher -h will show you some examples) and see
what it outputs. The script will tell you about anything that should be
installed and isn't, shouldn't be installed and is, and any packages that
are listed in both files. Which I consider a configuration error. And now to
an added feature, the local package list. This is the last list checked and
allows you to do host specific overrides. If, for example, you banned GCC
on all your servers you'd add it to the prohibited package list. And if you
had a build host that required GCC you'd have to either leave the check off
or customise the file for that one machine. By adding GCC to a local
packages file you can override that one package on that single host.
PkgWatcher was designed to run under Nagios but works just as well as an ad-hoc command line tool. Although without centralised management, keeping the required and prohibited lists up-to-date and in sync could become a hassle. And now some notes, it understands the RPM and DPKG packaging systems (and it's pretty easy to add additional ones), it's written in pure-perl (so it's easy to move around) and it's quite forgiving. If a package isn't on its required or prohibited lists then it does nothing about it. This is both because I'm pragmatic (a good deployment strategy is a better solution to keeping hundreds of machines in check) and because the environment I've written it for has a lot of legacy systems. And being overly strict means you never gain any ground.
What's next for it? It'll soon be plugged in to the configuration templating system we use to get automatic package checking based upon the services defined for that type of host (with package list generation based on the declared OS type).
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/01/02 22:42 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 17 Oct 2006
Check Disk... Checker?
You start off with a couple of partitions. You add a MySQL instance and
put it on a new logical volume. You break its logging out to a different
volume group for performance reasons. You take a snapshot for query
tuning and mount that. You add a chunk of disk for a
short experiment you were going to try... thanks to legacy,
laziness and easy to use LUNs you eventually end up with more mount
points than you know what to do with. And at the worst possible moment
one of them will fill and you'll discover you forgot to add it to Nagios for monitoring. Or you inherit a bundle
of crack fueled servers that have been "evolved" and never gifted with
decent monitoring.
The check_disk_checker.pl script was written to help find mount points that you're not monitoring. It scans through your local Nagios NRPE config files, looks at your current mount points, and complains about any mounted partitions that are not being checked according to the local NRPE configuration files. Of course there is nothing to say that what you have locally is what the remote Nagios is polling but that's outside the scope of this post.
check_disk_checker.pl shells out and grabs a list of all mounted partitions. It then
pulls a list of check-disk lines out of any config files
matching nrpe*.conf or nrpe*.cfg (our local naming scheme) that live
in /etc/nagios. It then extracts the partitions each one checks (it
grabs the value following a -p argument) and complains if it
doesn't find a check for each mounted partition. The script can be run
under Nagios as a plugin or stand-alone for help controlling a legacy
system.
It also plays well with the Linux Check Mounted Disks Nagios Plugin...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/10/17 00:04 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 01 Oct 2006
Linux Check Mounted Disks Nagios Plugin
For my own use as much as anyone elses... One of the problems that's
haunted me at least once per company I've worked at as a tech is "the
disappearing partition". It's there, it's accessible, and it should be
persistent across boots. But it isn't! The machine reboots and
then you discover that the database partition is no longer visible.
The check mounted
disks Nagios plugin looks at the mounted partitions and compares them
to what's in /etc/fstab (minus a couple of things like cd
drives, floppy disks, swap partitions etc). And warns if there are any
discrepancies that'll bite you on a reboot. It also round trips and makes
sure what /etc/fstab thinks is mounted is actually there.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/10/01 23:01 | /tools/commandline | Permanent link to this entry | This entry + same date
Fri, 29 Sep 2006
ps Problems
ps is an incredibly flexible command but it also has a
checkered maintenance history in the Linux world. Yesterday I needed to
output just the username, the command and any arguments passed to it. And
it was hell. After reading through the man page a couple of times I settled
on the following: ps -e -o user,args. But this doesn't
work.
It shows the command and the full arguments but it trunks the username
at 8 characters (which doesn't help with things like exim on Debian - which
has a username of Debian-exim). I then tried switching the order around to
see what happens, and was surprised when ps decided to
truncate the command and argument details at a seemingly arbitrary (but
consistant) point. GAH!
In the end I was pointed at a more correct, but ugly and not obvious
from the man page, answer; supply a width to the arguments. So to show
multiple fields in ps and not have one of them truncated you need a command
like this: ps -e -o user:20,args. And a smart friend like
Paul.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/29 08:03 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 03 Sep 2006
Open Ports Nagios Check
A machine should run a defined set of ports, if any of them are not
listening you've got a problem. If any others are open then you've
potentially got an even bigger problem. The Check Open
Ports Nagios Check accepts a list of IPv4 TCP and UDP ports and
reports if any of the expected ones go away or any others are detected
as listening.
This also partially scratches one of my own itches, I've had a couple of daemons (MySQL in particular) start after a package upgrade without my knowing it. With this script and a little cron it won't happen again. It's probably worth mentioning that while this script is built to run within Nagios it will work stand-alone.
Note: this script is more for detecting misconfigurations than for
security. Most root kits mask the ports they've opened so they won't appear
through netstat, which this command uses.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/03 12:56 | /tools/commandline | Permanent link to this entry | This entry + same date
Fri, 01 Sep 2006
Debian Packages Requiring Updates - Via Nagios
I've recently needed a way to see, via the Nagios web front end, which
Debian machines need their packages updating. So I wrote the
check_debian_updates.sh Nagios plugin. This is the initial release
(which hasn't been hit too hard yet) so be careful about deploying it
anywhere but your testing environment for now. I've played with it in my
small test environment and it seems to work so feel free to have a look
at it. I'll be stressing it, and possibly tidying the code up a little,
next week.
In its basic operation, the script just reports how many packages (if any)
need updating and returns a CRIT or a WARN to Nagios based upon your
thresholds. If you call the script with a -v it will also
output the name of all packages that need updating. Which may consume a
lot of Nagios front end screen real estate. Due to it running
apt-get update it needs some root privileges. I'll be
setting up sudo to let the Nagios user run this as root with no password
for both the apt-get update and apt-get upgrade
-s (note the '-s' for simulation.) And only for those!
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/09/01 23:57 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 15 Jul 2006
HTML Tidy, FireFox Envy and the Command Line
Tidy is a great little HTML lint
tool, that goes a lot further than the W3C Validator, but it requires
you to remember to run it. The FireFox HTML Validator
extension uses tidy and the FireFox status bar at the bottom of the
screen to show you tidy output from the current page.
This extension removes the need to run tidy by hand, you get it for free on every page you visit, but it does mean you need to visit any pages you want to run tidy against once you get spoiled by its output. And you will! The biggest annoyance with the tidy command line tool the requirement that pages must be on disk, not on a live website, which isn't what I wanted. The tidy_page.sh command line script is my little wrapper that could; it wraps tidy and wgets the pages for you. It then displays the results and removes the temp file so no traces remain on disk.
With the addition of a little wrapping script around this and I've now got a daily lint check on my main webpages. And you can have one too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/07/15 15:00 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 28 Jun 2006
Short File Copy Command Line
Making a backup copy of a file is a pretty common thing to do (although you
should be using RCS for a lot of these...). If you're using a machine with a
GUI then copy and pasting the file name twice, with an extension on the
end, is pretty simple. If you're either a keyboard jockey or without a
mouse you can make your life easier with these two short cuts:
# make a copy of file. add the .old extension
cp filename{,-old}
# overwrite the old version with the new version
cp filename{-old,}
# make a backup of a file deep in a directory tree. The copy
# is in the remote directory, not the working one.
cp /var/www/www.unixdaemon.net/htdocs/index.html{,-old}
It's not a huge saving until you find yourself with long filenames, working on a machine without a GUI but it is a useful trick to know.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/06/28 18:47 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 08 Mar 2006
IRC Quotes From bash.org - Command Line Tool
One of my guilty pleasures is reading through IRC quotes. I hate to think how much time I've spent reading my way through bash.org and qdb.us. While playing with
Template::Extract today I found myself needing a simple,
structured site to experiment with. And it resulted in the bash_quotes command line
tool.
The script is pretty simple, if you call it without an argument it gets
the quotes from the "Latest" page. If you call 'bash_quotes
-r' then it'll fetch a random set. You can then run it numerous
times to fill any amount of spare time. Which may not be a good thing.
I'm quite impressed with Template::Extract, it's a very neat way of
extracting data from websites without doing a lot of state retention in
your own code.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/03/08 21:11 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 18 Feb 2006
frdns.pl - Forward and Reverse DNS Lint
The frdns.pl forward and
reverse DNS checking script is one of those little mistake catchers
that allow you to work with a safety net. In this case it checks that
your deployed forward and reverse DNS records are present and correct; it
checks the results from real DNS queries, not by zone file parsing.
frdns.pl accepts a CIDR range and polls each IP for a reverse DNS record. If it gets one it'll try to forward resolve the name and compare the two results. If the forward record is missing, or the two parts don't match, it'll print the problem. You can supply a flag to make it only display valid or broken records and with a little bit of shell wrapping it (and iterating through your networks) you've got a nice cronjob that'll give you a bit more confidence that your DNS is clean.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2006/02/18 14:19 | /tools/commandline | Permanent link to this entry | This entry + same date
Copyright © 2000-2005 Dean Wilson
