Wed, 30 Sep 2009
Rake - surprisingly enjoyable
I've never really liked make files, I don't think I've ever had to write
enough C to really appreciate (or just tolerate) them, so I was a little
dismissive of Rake - and I was
mostly wrong.
Now we're adding a new member to the systems team I've been doing a lot
of thinking about our tool chain - what knowledge assumptions it makes,
which parts are still more manual than I'd like and where the tool chain has
gaps (this is the most annoying one for me) and rake seemed like a
potential addition to encode some of that process knowledge in to a tool.
I've only added little rakefiles here and there but they do make certain
tasks nicer (plus I like the inline descs).
I've not yet worked out any general rules for when to use a shell script and when to use rake but if nothing else it's helping me spend some time on my ruby skills. The best rake starting points I found were Martin Fowlers rake article and the rake release notes.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/30 21:48 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 01 Jul 2009
dstat - a window to your system
When it comes to Unix diagnostics I was raised the old fashion way, with
iostat, vmstat and similar tools. However times change and tools evolve.
dstat, while not as
comprehensive as using all the tools one by one, provides a wide range
of system performance details in an easy to use package.
While it's useful enough in its default state there is even more
functionality lurking just below the surface. To see which other modules
are available (but are not enabled by default) run dstat -M
list. To add an extra module to the output use a command like
this one: dstat -a -M topmem -M topcpu
As part of my growing use of the tool I've started to write my own little dstat plugins. I was pleasantly surprised at how easy they were to write and deploy even with my basic python skills. While the memcached plugin was a proof of concept I've not needed much I've found the process count plugin to be very handy.
dstat is becoming one of the overview tools I use when investigating performance issues and it's worthy of a place in your toolbox too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/01 21:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Mon, 09 Mar 2009
Puppet Scripts - extract-report-issues
I spent a little while digging through the default puppet log types the
other day and after reading through a batch of activity logs I whipped up
extract-report-issues,
a script that can be run on the command line (or daily via cron) and
displays a list of errors and warnings from the specified glob of hosts and
log files. By default it does all hosts for the current day, we've got it
running nightly so we can work through the issues each morning. It's worth
noting that sometimes in the output the same failure occurs more than
once. This is because puppet retries certain operations - such as retrieving
a resource.
There is actually a lot of useful information in the puppet reports. To start with I've added a todo item for a script that notes persistent errors (the same issues over two or three runs) that I'll hopefully get to this month. Maybe.
If you're running puppet in production you owe it to yourself to turn on reporting and set up some processes around it. While puppet makes it easy to perform action at a distance you still need to close the loop somehow.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 20:57 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 03 Feb 2009
Simple, Single Document Bookmarks in vim
I like vim, I think it's a great editor
worth investing time and effort in to learning but I also think it's one
of the most horrible things to watch an inexperienced user typo his way
through while you're urgently waiting for them to finish the damn edit.
My favourite one this week (and it's only Tuesday) is looking for
probably unique phrases that you can later search for to return to a
specific part of a document.
In an attempt to stop my laptop getting any more back of the head shaped
dents in it from when I've failed to restrain myself I thought I should
point out a much simpler way of doing this. Once you're at the part of a
document you want to return to press m<letter>. This
sets a mark. To return to it press '<letter>. That's it.
No more pasting in chunks of a string hoping it only occurs once in the
damn document. If you need to mark a couple of locations then fine just use
different letters to set and return to the places you want. And save me
sending another laptop back in for warranty.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/03 22:05 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 14 Jan 2009
Soon to be With Added Git?
Despite setting up my own
gitweb
install I'm still not using git regularly enough to be comfortable with it
so today I went through the Peepcode Press Git Internals book/PDF. While
the diagrams and details of what happens under the cover are useful it's
the wrong level for me as a basic user. To ease myself in to the move
from subversion for some of my personal projects I found Git Magic to be more
useful.
I know git requires a mental shift and it's a very complex and powerful tool but for my own needs I'll probably never use more than 10% of its capabilities. Unfortunately most of the projects I use and need to submit patches to have switched - so I'll be a happy sheep and go along for the ride. Even if it turns out to be a roller coaster.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/14 18:14 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 06 Jan 2009
Diffing Files Over Multiple Servers - rd-differ
Adhoc changes are a very bad thing in many ways, one of the worst is how
often they are not fully implemented across all the servers or even
pulled back to staging. In an attempt to sanity check the config files when
we have to make these little hacks I oddly-proudly present - rd-differ. A tool for
diffing config files over multiple machines.
The idea is simple, you tell it the file or directory you're interested
in, specify a single machine as the baseline and then specify a number of
others as the machines to check against it. A sample invocation looks like this
rd-differ /etc/apache2 10.10.100.111 10.10.100.112
10.10.100.113 and the output is show as a diff.
The files are rsynced down using ssh so your usual keys will work and while the normal output is that of the raw diff it's very easy to wrap the results and add other checks on top of it. The shell's not written to be very defensive (unusual for me) but the code is short enough that it's worth the compromise.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/06 18:26 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 08 Nov 2008
Rebooting Via Proc and the magic sysreq key
You know what the best way to start the day is? I'm pretty sure that it
doesn't include a production web server putting its file systems in to
read only mode. When this happens most local commands don't work - init,
shutdown, telnit and reboot all stop being useful and you have to resort
to desperate measures... and here's the desperate
measure of the day.
First, check that your system supports the magic sysreq key -
$ cat /proc/sys/kernel/sysrq
1 # nonzero is good
Now you know you have the power to destroy your system through a single incorrect character, have a look at the Redhat Sysrq command reference (you want the 'sysrq' section). We tried to make it sync the disks and reboot - your requirements may vary.
root@web02:~# echo s > /proc/sysrq-trigger
root@web02:~# echo b > /proc/sysrq-trigger
# machine reboots
As techniques go this one's a little obscure but it's very useful in the right circumstances.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/11/08 12:25 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 23 Aug 2008
Nagios Service and Hosts stats - Graphed in Munin
We've been hitting some load issues on one of our monitoring
machines recently and while it looks like the munin graph generation
is the culprit we also decided to keep an eye on how many services and
hosts Nagios was checking.
One of the downsides of having a very automated server deployment system is how easy it is to suddenly find yourself with an extra dozen hosts you no longer really need. While each check is quite small and quick, add up the frequent runs and multiply it by a reasonable number of servers and you can soon hit problems.
So as a first step towards keeping an eye on those numbers we now have a munin Nagios hosts plugin and a munin Nagios services plugin that show the total number of hosts and services monitored and the states those resources are in.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:20 | /tools/commandline | Permanent link to this entry | This entry + same date
Nagios Checks - Validate HTML and Validate Feed
As part of my ongoing attempt to stop myself from silently making
mistakes (I don't so much mind the ones I notice) I've added another
couple of Nagios
Plugins. This time validate_feed
and validate_html.
As both of these checks call out to an external, third party resource, if you use them be sure to tweak your Nagios polling interval down to a respectful level.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:11 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 14 Aug 2008
Filter syslog logs with syslogslicer
While digging through a pile of syslog log files recently I needed
something a little more data format aware than pure grep. So I present the
first version of syslogslicer
- a simple perl script that knows a little bit about the syslog log file
format.
# some example command lines
syslogslicer -p cron -f program,message /var/log/syslog
# print the program and message for all lines with program 'cron'
syslogslicer -p cron -m hourly /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
syslogslicer -p cron -m hourly -s 20080810100000 -e 20080810123000 /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
# between 20080810100000 and 20080810123000
syslogslicer allows you to filter the output by matching text in the program or log message, only print certain output fields and do basic time based filtering. If you've ever wanted to see all the logs raised by postfix with the word 'database' in them between 10 and 11 am then this might be the tool for you.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 12:28 | /tools/commandline | Permanent link to this entry | This entry + same date
Nagios - Check Proxy Check
"This script retrieves a URL via a specified proxy server and
alerts (using the standard Nagios conventions) if the request
fails."
We're running a couple of services through a proxy server for a number of good, and to be honest a couple of not so good but mandated, reasons. The Check Proxy Check Nagios Plugin ensures that if the proxy goes down in a way that stops us pulling pages through it we know.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 09:30 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 13 Aug 2008
Nagios Disk Check - Mountpoint or Filesystem?
If you mount filesystems under a specific mount point, and monitor
them with Nagios, then be sure
you understand what happens if the underlying file system goes away.
With:
/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -p /a_mount_point
you'll get the value from the containing file system. In this case
/. If you'd rather know that your chosen mount point has
actually gone away, and that you're no longer checking what you thought
you were, then add the -E option to the command. This will
turn on exact path matching and catch that kind of error.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/13 21:54 | /tools/commandline | Permanent link to this entry | This entry + same date
Testing the 'Net isn't there with Nagios
We've recently had to deliberately disable some machines this week to
ensure they can't connect out to the internet - we're building testing
versions of some of our more restricted secure environments and this is
one of the steps.
It was actually easier to do with IPTables than I thought (mostly because I didn't have to do it - my co-worker did) but once the work was done we needed to ensure it didn't accidently get broken so that networking was functional again. And yes that's an odd thing to type. So naturally we turned to Nagios and so, for my own memory as much as anything else, here is the check we're using:
# put this in the machines nrpe config file.
/usr/lib/nagios/plugins/negate -t 30 "/usr/lib/nagios/plugins/check_http -w 5 -c 10 -H www.google.com -u /"
In the Nagios 'Status Information' field you'll get a message that
looks like this - CRITICAL - Socket timeout after 10
seconds - but the check returns the correct error code so it's
all green.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/13 21:50 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 12 Aug 2008
Yumdpkg-provides
I've never really felt as proficient with apt and dpkg as I did with RPM.
There always seems to be another option I've never seen before.
Luckily there are also big holes in my knowledge of yum to make me feel well
rounded.
After reading yum options you may not know exist and spending a while puzzling out how to get the same results in Debian (apt-file seems to be the closest fit but I never got the invocation right) I decided to write dpkg-provides.
It's not packaged, doesn't have a manpage, requires the network and isn't integrated with the existing tools. At least I know how I'd get the information now - from the web. Who'd thought it?
Note: it's actually quite simple to work out which package provides a file
that you've got installed locally (dpkg -S '*/df') - it's more
of a pain to probe packages you don't have installed.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/12 15:13 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 08 Jul 2008
Dear Lazyweb - Command Line YSlow!
The title pretty much says it all, I'd like a command line version of YSlow! (what is it with
Yahoo and !s) that I can run from cron and import in to a nice
spreadsheet for trending and site comparisons.
I don't have XUL on my list of things to play with so I'll give it a couple of months and watch someone else implement it. Hopefully.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/07/08 19:59 | /tools/commandline | Permanent link to this entry | This entry + same date
Mon, 25 Jun 2007
Navigating Commented Config Files
The current trend with config files is to fill them with comments (let's
ignore the fact this isn't a substitute for documentation) and while this
is helpful watching people arrow through them line by line looking for
active options drives me nuts.
If you're using vim (as all good people
do ;)) you can jump from uncommented directive to uncommented directive
with /^[^#] as a search. Pressing n will then
move you to the next uncommented option. And save me from pulling out those
precious few hairs I have left.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/25 21:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 03 Jun 2007
Nagios - Simple Trender
Continuing the release of my Nagios code - here's my Nagios
Simple Trender. It parses Nagios logs and builds a horizontal barchart
for host outages, service warnings and criticals. It's nothing fancy (and
the results are a little unpretty) but it does make the attention seeking
services and hosts very easy to find.
While the tool isn't that technically complex I've found it useful in justifying my time on certain parts of the infrastructure. Being able to show how bad NTP is for example (we had 216 NTP sync problems last month, this month we had 36; and most of those are one machine with a bad clock) on a very simple chart makes it easier to get buy in from above. And next month you can show them how much of a positive impact the work had.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 10:29 | /tools/commandline | Permanent link to this entry | This entry + same date
The Nagios Tag Cloud
We use the Nagios monitoring system
at work (in fact we use four installs of it for physically isolated
networks) and while it's damn useful (and service checks are easy to
create or extend) it's a little lacking in higher level trending and
visualisation tools. Well, at least the very old version we run suffers
from this.
Thankfully I work for a company that invests time in its core tools. Over the last couple of hackdays I've written two small scripts for parsing Nagios logfiles and presenting the information in a different, slightly more grouped way. The first of these is the Nagios TagCloud - which has a very descriptive name :)
When invoked (I typically use nagiosclouds.pl /log/files/*.log >
/webdir/nagios_tagcloud.html from a cronjob) it'll run through the log
files and produce a HTML page containing 3 tag clouds, one for host
outages, one for service warnings and one for service criticals. Tag clouds
don't suit everyones work style but I came away from running ours with a
couple of action points so I think they're useful enough to glance at once
a month.
I should note the perl module that generates the tag cloud is Leon Brocards HTML::TagCloud and the CSS was graciously given to me by Alex Monney after he burned his eyes looking at my first version.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 10:08 | /tools/commandline | Permanent link to this entry | This entry + same date
The Perk of $HOME - .bash_logout
It's not a well kept secret but I'm still surprised by how many people have
never encountered .bash_logout. Its purpose is pretty simple,
if you use the BASH shell it'll be executed when you log out (see, a well
named file!)
So what's it for? Well, I use mine to invalidate any sudo
sessions I've got open (sudo -k), clear the
screen -in case it's a local session - and nuke a history file or two.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/06/03 09:47 | /tools/commandline | Permanent link to this entry | This entry + same date
Sun, 06 May 2007
df Output Ordering
Sometimes questions come up that you know you should know the answer to but
you just don't. My recent one was "how does df choose the
output order?" The man page doesn't mention the logic behind it and a quick
strace shows it pulls its data from /proc/mounts
(which you'd expect) and returns the output in the same order. So logically
the question becomes how does /proc/mounts order things?
It's not exactly an important question but I can see how this ends - and it involves source code.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2007/05/06 11:46 | /tools/commandline | Permanent link to this entry | This entry + same date
Copyright © 2000-2005 Dean Wilson
