Wed, 30 Sep 2009
Rake - surprisingly enjoyable
I've never really liked make files, I don't think I've ever had to write
enough C to really appreciate (or just tolerate) them, so I was a little
dismissive of Rake - and I was
mostly wrong.
Now we're adding a new member to the systems team I've been doing a lot
of thinking about our tool chain - what knowledge assumptions it makes,
which parts are still more manual than I'd like and where the tool chain has
gaps (this is the most annoying one for me) and rake seemed like a
potential addition to encode some of that process knowledge in to a tool.
I've only added little rakefiles here and there but they do make certain
tasks nicer (plus I like the inline descs).
I've not yet worked out any general rules for when to use a shell script and when to use rake but if nothing else it's helping me spend some time on my ruby skills. The best rake starting points I found were Martin Fowlers rake article and the rake release notes.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/30 21:48 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 23 Sep 2009
Simplifying File Permissions in Puppet Manifests
I've been a user of Puppet for about three years now and
while on a recent dig in to some of my older classes it was a little
embarrassing to see lots of file types used like this:
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
owner => whi,
group => whi,
mode => 644
}
Luckily as we get more experienced with a tool we can often go back and
improve on the first steps. By using an explicit File { settings
} inside a class you can assign a sensible set of defaults to all
the instances of the same type that lack overriding settings. So we can
shorten the previous example to -
File {
owner => whi,
group => whi,
mode => 644
}
file { "/srv/whi/maps":
ensure => present,
source => "puppet://$servername/whi/maps.conf"
}
file { "/srv/whi/elocs":
ensure => present,
source => "puppet://$servername/whi/eloc.conf"
}
While this isn't a huge win in raw characters typed (although in longer manifests they start to mount up) it does move all the common settings in to a single location (keeping us clear of DRY violations) and it leaves only the differences between file type definitions.
You can also apply those kind of settings (such as Exec { path =>
"path:list" } at the server level by including them in a top level
file and then overriding them as needed in each module. If you do this then
you need to be aware that any declared type that doesn't override it gets
the global setting, which can lead to the odd action from afar head
scratching.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/23 22:16 | /tools/puppet | Permanent link to this entry | This entry + same date
Thu, 17 Sep 2009
Stand Alone Puppet
While Puppet can be used to manage large, complex
environments it's also a useful tool at the lower end of the spectrum.
Using just the puppet executable and a small inline class or
two you can write very useful manifests in only a handful of lines.
class build-host {
package { "build-essential": ensure => installed }
package { "subversion": ensure => installed }
file { "/home/dwilson/repos/":
ensure => directory,
owner => dwilson,
group => dwilson,
}
}
node default {
include build-host
}
To invoke the class you just run puppet -v build-host.pp.
It's also worth pointing out the node name of default. This saves you
manually changing the manifest whenever you move to another machine.
While it wouldn't be hard to replace the above example with a shell
script, by using puppet you can easily access the built-in abstractions
(which package manager to use, how should you add users) and remove a lot
of scaffolding code. And then when you're done you can promote the class to
your managed infrastructure.
I've used this to bootstrap provisioning servers (why should the provisioning host be the only machine that wasn't provisioned?), test small but annoying new classes on scratch servers and I'm currently working on integrating it with a small subversion backup testing project in my spare time at work (so very slowly).
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/09/17 21:33 | /tools/puppet | Permanent link to this entry | This entry + same date
Wed, 01 Jul 2009
dstat - a window to your system
When it comes to Unix diagnostics I was raised the old fashion way, with
iostat, vmstat and similar tools. However times change and tools evolve.
dstat, while not as
comprehensive as using all the tools one by one, provides a wide range
of system performance details in an easy to use package.
While it's useful enough in its default state there is even more
functionality lurking just below the surface. To see which other modules
are available (but are not enabled by default) run dstat -M
list. To add an extra module to the output use a command like
this one: dstat -a -M topmem -M topcpu
As part of my growing use of the tool I've started to write my own little dstat plugins. I was pleasantly surprised at how easy they were to write and deploy even with my basic python skills. While the memcached plugin was a proof of concept I've not needed much I've found the process count plugin to be very handy.
dstat is becoming one of the overview tools I use when investigating performance issues and it's worthy of a place in your toolbox too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/01 21:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Mon, 09 Mar 2009
Puppet Scripts - extract-report-issues
I spent a little while digging through the default puppet log types the
other day and after reading through a batch of activity logs I whipped up
extract-report-issues,
a script that can be run on the command line (or daily via cron) and
displays a list of errors and warnings from the specified glob of hosts and
log files. By default it does all hosts for the current day, we've got it
running nightly so we can work through the issues each morning. It's worth
noting that sometimes in the output the same failure occurs more than
once. This is because puppet retries certain operations - such as retrieving
a resource.
There is actually a lot of useful information in the puppet reports. To start with I've added a todo item for a script that notes persistent errors (the same issues over two or three runs) that I'll hopefully get to this month. Maybe.
If you're running puppet in production you owe it to yourself to turn on reporting and set up some processes around it. While puppet makes it easy to perform action at a distance you still need to close the loop somehow.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 20:57 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 04 Mar 2009
First Steps in Github
I finally decided to set up and start using a github account and my early
impressions are that it's quite slick and very userfriendly.
Apart from an annoyance where I couldn't see my pushes for a little while (I think I fell afoul of some caching) setting up an account and adding new repos was simple. Pushing from my actual dev machine just worked and I've now been bitten by the github bug. I feel the need to push code...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/04 22:22 | /tools/online | Permanent link to this entry | This entry + same date
Mon, 02 Mar 2009
Mozilla Ubiquity - Puppet Types
I like Ubiquity. It
puts a lot of the sites I used on a regular basis close to hand without
making me dig through my bookmarks (or del.icio.us account). In a
small burst of productivity, and to avoid real work, I decided to put a
command together for the Puppet Type docs at Reductive Labs.
If you have the Ubiquity plugin installed you should be able to install a copy of the command from the Ubiquity Puppet Types Command page. Writing a simple command like this one was surprisingly easy (especially using command-editor) although I don't (yet) know enough jQuery to add some of the other features I want.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/02 22:55 | /tools/firefox | Permanent link to this entry | This entry + same date
Tue, 03 Feb 2009
Simple, Single Document Bookmarks in vim
I like vim, I think it's a great editor
worth investing time and effort in to learning but I also think it's one
of the most horrible things to watch an inexperienced user typo his way
through while you're urgently waiting for them to finish the damn edit.
My favourite one this week (and it's only Tuesday) is looking for
probably unique phrases that you can later search for to return to a
specific part of a document.
In an attempt to stop my laptop getting any more back of the head shaped
dents in it from when I've failed to restrain myself I thought I should
point out a much simpler way of doing this. Once you're at the part of a
document you want to return to press m<letter>. This
sets a mark. To return to it press '<letter>. That's it.
No more pasting in chunks of a string hoping it only occurs once in the
damn document. If you need to mark a couple of locations then fine just use
different letters to set and return to the places you want. And save me
sending another laptop back in for warranty.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/03 22:05 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 14 Jan 2009
Soon to be With Added Git?
Despite setting up my own
gitweb
install I'm still not using git regularly enough to be comfortable with it
so today I went through the Peepcode Press Git Internals book/PDF. While
the diagrams and details of what happens under the cover are useful it's
the wrong level for me as a basic user. To ease myself in to the move
from subversion for some of my personal projects I found Git Magic to be more
useful.
I know git requires a mental shift and it's a very complex and powerful tool but for my own needs I'll probably never use more than 10% of its capabilities. Unfortunately most of the projects I use and need to submit patches to have switched - so I'll be a happy sheep and go along for the ride. Even if it turns out to be a roller coaster.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/14 18:14 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 06 Jan 2009
Diffing Files Over Multiple Servers - rd-differ
Adhoc changes are a very bad thing in many ways, one of the worst is how
often they are not fully implemented across all the servers or even
pulled back to staging. In an attempt to sanity check the config files when
we have to make these little hacks I oddly-proudly present - rd-differ. A tool for
diffing config files over multiple machines.
The idea is simple, you tell it the file or directory you're interested
in, specify a single machine as the baseline and then specify a number of
others as the machines to check against it. A sample invocation looks like this
rd-differ /etc/apache2 10.10.100.111 10.10.100.112
10.10.100.113 and the output is show as a diff.
The files are rsynced down using ssh so your usual keys will work and while the normal output is that of the raw diff it's very easy to wrap the results and add other checks on top of it. The shell's not written to be very defensive (unusual for me) but the code is short enough that it's worth the compromise.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/06 18:26 | /tools/commandline | Permanent link to this entry | This entry + same date
Sat, 08 Nov 2008
Rebooting Via Proc and the magic sysreq key
You know what the best way to start the day is? I'm pretty sure that it
doesn't include a production web server putting its file systems in to
read only mode. When this happens most local commands don't work - init,
shutdown, telnit and reboot all stop being useful and you have to resort
to desperate measures... and here's the desperate
measure of the day.
First, check that your system supports the magic sysreq key -
$ cat /proc/sys/kernel/sysrq
1 # nonzero is good
Now you know you have the power to destroy your system through a single incorrect character, have a look at the Redhat Sysrq command reference (you want the 'sysrq' section). We tried to make it sync the disks and reboot - your requirements may vary.
root@web02:~# echo s > /proc/sysrq-trigger
root@web02:~# echo b > /proc/sysrq-trigger
# machine reboots
As techniques go this one's a little obscure but it's very useful in the right circumstances.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/11/08 12:25 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 04 Sep 2008
Ubiquity - More Than Just Shiny Chrome
While Google Chrome has been getting all the press coverage recently Ubiquity, from Mozilla Labs, is where all the interesting action seems to be happening.
Ubiquity ticks all the boxes for me, it's a simple, easy to use idea, that'll save me time. It's easily extensible and already has a huge community of people working, enhancing and just trying new things with it. All the things I've come to expect from Firefox and the Mozilla using community.
I personally think this is an important distinction to make - while Google Chrome is a new browser with some great ideas (and a quickly revised EULA) FireFox is a proven, Free platform that encourages extension and has a track record of doing the right thing.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/09/04 19:58 | /tools/firefox | Permanent link to this entry | This entry + same date
Sat, 23 Aug 2008
Nagios Service and Hosts stats - Graphed in Munin
We've been hitting some load issues on one of our monitoring
machines recently and while it looks like the munin graph generation
is the culprit we also decided to keep an eye on how many services and
hosts Nagios was checking.
One of the downsides of having a very automated server deployment system is how easy it is to suddenly find yourself with an extra dozen hosts you no longer really need. While each check is quite small and quick, add up the frequent runs and multiply it by a reasonable number of servers and you can soon hit problems.
So as a first step towards keeping an eye on those numbers we now have a munin Nagios hosts plugin and a munin Nagios services plugin that show the total number of hosts and services monitored and the states those resources are in.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:20 | /tools/commandline | Permanent link to this entry | This entry + same date
Nagios Checks - Validate HTML and Validate Feed
As part of my ongoing attempt to stop myself from silently making
mistakes (I don't so much mind the ones I notice) I've added another
couple of Nagios
Plugins. This time validate_feed
and validate_html.
As both of these checks call out to an external, third party resource, if you use them be sure to tweak your Nagios polling interval down to a respectful level.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/23 14:11 | /tools/commandline | Permanent link to this entry | This entry + same date
Thu, 14 Aug 2008
Filter syslog logs with syslogslicer
While digging through a pile of syslog log files recently I needed
something a little more data format aware than pure grep. So I present the
first version of syslogslicer
- a simple perl script that knows a little bit about the syslog log file
format.
# some example command lines
syslogslicer -p cron -f program,message /var/log/syslog
# print the program and message for all lines with program 'cron'
syslogslicer -p cron -m hourly /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
syslogslicer -p cron -m hourly -s 20080810100000 -e 20080810123000 /var/log/syslog
# all fields for all lines with program 'cron' and message 'hourly'
# between 20080810100000 and 20080810123000
syslogslicer allows you to filter the output by matching text in the program or log message, only print certain output fields and do basic time based filtering. If you've ever wanted to see all the logs raised by postfix with the word 'database' in them between 10 and 11 am then this might be the tool for you.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 12:28 | /tools/commandline | Permanent link to this entry | This entry + same date
Nagios - Check Proxy Check
"This script retrieves a URL via a specified proxy server and
alerts (using the standard Nagios conventions) if the request
fails."
We're running a couple of services through a proxy server for a number of good, and to be honest a couple of not so good but mandated, reasons. The Check Proxy Check Nagios Plugin ensures that if the proxy goes down in a way that stops us pulling pages through it we know.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/14 09:30 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 13 Aug 2008
Nagios Disk Check - Mountpoint or Filesystem?
If you mount filesystems under a specific mount point, and monitor
them with Nagios, then be sure
you understand what happens if the underlying file system goes away.
With:
/usr/lib/nagios/plugins/check_disk -w 15% -c 10% -p /a_mount_point
you'll get the value from the containing file system. In this case
/. If you'd rather know that your chosen mount point has
actually gone away, and that you're no longer checking what you thought
you were, then add the -E option to the command. This will
turn on exact path matching and catch that kind of error.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/13 21:54 | /tools/commandline | Permanent link to this entry | This entry + same date
Testing the 'Net isn't there with Nagios
We've recently had to deliberately disable some machines this week to
ensure they can't connect out to the internet - we're building testing
versions of some of our more restricted secure environments and this is
one of the steps.
It was actually easier to do with IPTables than I thought (mostly because I didn't have to do it - my co-worker did) but once the work was done we needed to ensure it didn't accidently get broken so that networking was functional again. And yes that's an odd thing to type. So naturally we turned to Nagios and so, for my own memory as much as anything else, here is the check we're using:
# put this in the machines nrpe config file.
/usr/lib/nagios/plugins/negate -t 30 "/usr/lib/nagios/plugins/check_http -w 5 -c 10 -H www.google.com -u /"
In the Nagios 'Status Information' field you'll get a message that
looks like this - CRITICAL - Socket timeout after 10
seconds - but the check returns the correct error code so it's
all green.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/13 21:50 | /tools/commandline | Permanent link to this entry | This entry + same date
Tue, 12 Aug 2008
Yumdpkg-provides
I've never really felt as proficient with apt and dpkg as I did with RPM.
There always seems to be another option I've never seen before.
Luckily there are also big holes in my knowledge of yum to make me feel well
rounded.
After reading yum options you may not know exist and spending a while puzzling out how to get the same results in Debian (apt-file seems to be the closest fit but I never got the invocation right) I decided to write dpkg-provides.
It's not packaged, doesn't have a manpage, requires the network and isn't integrated with the existing tools. At least I know how I'd get the information now - from the web. Who'd thought it?
Note: it's actually quite simple to work out which package provides a file
that you've got installed locally (dpkg -S '*/df') - it's more
of a pain to probe packages you don't have installed.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/08/12 15:13 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 09 Jul 2008
Amazon^WLoveFilm DVD Rentals
I've been a happy Amazon UK DVD rental customer for the last couple of
years. They've got a wide selection, the DVDs ship fast, come in
separate envelopes and in nice sturdy plastic cases. In nearly 200 DVDs
I've had three that were unplayable and only one that got lost in
transit - a replacement for which was sent the same day.
'Luckily' for me Amazons DVD rentals are now handled by LoveFilm. I had my first batch through last week and I'm less than impressed. All three films came in a single envelope (so I have to watch them all before I can sent them back for replacement), they are in flimsy paper containers and out of my first two one is broken - with a split straight through. This is not the best start I could have hoped for.
There's a reason Amazon gets a lot of my online cash - they provide a damn good service.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2008/07/09 18:52 | /tools/online | Permanent link to this entry | This entry + same date
Copyright © 2000-2005 Dean Wilson
