Fri, 03 Jul 2009
By Puppet or Package
At work we both build our own packages and use puppet to manage our
servers. While the developers package up their work in the systems team
we've moved more to deploying programs and their dependencies via
Puppet.
While it seems easier, and quicker, to do the pushing that way, at least for scripts, you lose the ability to track what's responsible for putting each file on the system. I'm probably already modelling the more complex parts of what would be in a package (such as services and cronjobs) in the module and thanks to Puppet I'm probably doing it in quite a portable way. Is this actually better than using packages? It's certainly easier than building complex packages but it quickly gets awkward when you start needing to deploy compiled binaries or apps with lots of moving parts.
For now my rule seems to be - use puppet for small, non-compiled, apps and package up anything with lots of dependencies or that needs to be compiled. How do you deploy your infrastructure scripts and supporting artifacts?
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/03 22:12 | /sysadmin | Permanent link to this entry | This entry + same date
Thu, 02 Jul 2009
EuroPython 2009 - Wrap up Post
Over the last week I've been up in Birmingham catching up with some old
friends and attending some talks at the little get together of around 450
Pythonistas that was EuroPython
2009.
This was my second Python conference. The first was PyCon 2008, which was so well organised (by many of the same team as this years EuroPython) that I was inspired to come back. And I wasn't disappointed. There were a lot of very good talks, some that have planted seeds that I'll have to come back and try to find the time to look at and some that showed me things I plan on using in the very near future (such as py.test).
The atmosphere was topnotch. Everyone seemed friendly, the speakers were approachable and after spending the evening with so many people working on so many things it was a pleasure to get back to the room and make sure I actually did something technical before bed.
It's a wonderful feeling to come away from a conference feeling motivated to try new technologies and all I need to do now is actually schedule some time actually write some Python code...
The organisers did a great job and I'll be back next year.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/02 21:13 | /events | Permanent link to this entry | This entry + same date
Wed, 01 Jul 2009
dstat - a window to your system
When it comes to Unix diagnostics I was raised the old fashion way, with
iostat, vmstat and similar tools. However times change and tools evolve.
dstat, while not as
comprehensive as using all the tools one by one, provides a wide range
of system performance details in an easy to use package.
While it's useful enough in its default state there is even more
functionality lurking just below the surface. To see which other modules
are available (but are not enabled by default) run dstat -M
list. To add an extra module to the output use a command like
this one: dstat -a -M topmem -M topcpu
As part of my growing use of the tool I've started to write my own little dstat plugins. I was pleasantly surprised at how easy they were to write and deploy even with my basic python skills. While the memcached plugin was a proof of concept I've not needed much I've found the process count plugin to be very handy.
dstat is becoming one of the overview tools I use when investigating performance issues and it's worthy of a place in your toolbox too.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/07/01 21:32 | /tools/commandline | Permanent link to this entry | This entry + same date
Wed, 03 Jun 2009
It's been Critical for how long?
Nagios has a wonderful 'duration' column in its web interface that's
always bemused me. At what point does a check being in a warning, or
even worse, a critical state stop being a problem worthy of head space
and start being normal operating procedure?
Checks can stay in an extended broken state for many reasons but they all seem to be symptoms of a larger problem. If it's a small thing then are you getting enough time to do housekeeping? If it's a big thing do you have enough business buy in to keep things running optimally? Are you monitoring the wrong thing? Is there even anything you can do to fix it? If not then maybe Nagios isn't the best place to put the monitoring, maybe a status report is a better place.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/06/03 07:55 | /sysadmin | Permanent link to this entry | This entry + same date
Wed, 08 Apr 2009
New Laptop? New job?
Is it just me or does everybody seem to go and buy a new laptop just
before they leave their current job? Is it the techie version of buying
new work shoes?
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/04/08 11:37 | /geekstuff | Permanent link to this entry | This entry + same date
2009 Q2 PiP
I've been quiet on the PiP front for a while now. While the day to day stuff
has kept me busy it hasn't exactly helped move my career along, I spend
most of my time doing things I already know how to do but with a little
twist on them.
In an attempt to stop myself from further stagnation I've put a short list of goals below that should be my bare minimum for the next three months.
- Attend one technical conference / training course
- Attend two technical user group meetings
- Read and review 3 books.
- Write and publish a Perl modules.
- Write and package a Python or Ruby module
- Write and publish a technical article.
- Create a personal Debian repo
- Create 3 Debian packages, at least one of which should contain other peoples code.
- Write 30 blog posts - at least 15 of which should be technical.
I'll do a follow up post at the end of the quarter so see how far I got.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/04/08 10:20 | /geekstuff | Permanent link to this entry | This entry + same date
Mon, 06 Apr 2009
Ruby DNS Testing - First Glance
DNS is one of those 'small config change here, errors a long way over there
later' technologies that always leaves me a little worried about the
knock on effect of my changes. As a simple, coarse, safeguard at work we
use Nagios to check that a canary record in each zone can be resolved
from each DNS server. It's far from a perfect solution but it does catch
some of the bigger errors and typos.
In order to beef up this safety net (and encourage me
to spend time using a language other than Perl) I've been
investigating some of the testing options available in ruby, namely
RSpec and Cucumber. I have to say the testing libraries themselves are
actually nice to use and easy to pick up even for a
non-rubyists like me. On the other hand I quickly developed a strong
dislike of the ruby resolv library that actually does the
DNS queries. The lack of decent tutorials or documentation for
anything beyond the very basic uses and the (to me) very awkward API
nearly had me running back to the safety of
Net::DNS
, a mature and widely used perl module. There is a ruby port that I'll
have a look at in the future
For the testing itself I started writing RSpec stories against my own DNS and found the API easy to use. Testing existing configs against local policies is simple - for example all our domains should list at least three name servers -
it "should have at least 3 NS records" do
@resolver = Resolv::DNS.new
@nameservers = @resolver.getresources( domain, Resolv::DNS::Resource::IN::NS)
@nameservers.should have_at_least(3).items
end
After doing some more scut work and testing other record types I moved onwards and upwards to Cucumber. I'm not sure I'd be able to gift someone else with writing the scenarios but they'll be easier to show business people than raw rspec stories. They'll also be very handy in migration meetings (an executable todo list), if it's not on the page it's not getting done.
Feature: Mass DNS Resolution
In order to present a consistent brand image
As a System Administrator
I want to ensure no domains point away from our main IP
Scenario Outline: Resolve a name to a number
Given a hostname of <hostname>
Then I should see the IP address 266.266.266.266
Examples:
| hostname |
| example.org |
| www.example.org |
Next time I work somewhere with ISO 27xxx compliance requirements I'll see if the controls can be written like this and have automatic verification run from them. Cucumber is a little wordy for my tastes but I can see where that could be a strength when presenting to the right audience (such as compliance auditors).
Next time we have a set of DNS migrations I'll be using at least one of these tools to write before and after test cases to ensure nothing gets missed or slips through the cracks. Once I've been through a couple I'll write up in more detail what we end up with.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/04/06 15:46 | /ruby | Permanent link to this entry | This entry + same date
Mon, 09 Mar 2009
Speakers - Always Repeat the Question
If someone in the audience asks a question that you plan on answering
then please repeat it, with your own wording, before you respond.
This gets us two things - the person asking probably won't have a mic so not everyone will hear what he said, they will when you repeat it. Secondly - by repeating with your own phrasing - you'll get basic confirmation that you understand what's being asked rather than answering the wrong question; which wastes everyones time and leaves the asker frustrated.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 21:06 | /geekstuff | Permanent link to this entry | This entry + same date
Puppet Scripts - extract-report-issues
I spent a little while digging through the default puppet log types the
other day and after reading through a batch of activity logs I whipped up
extract-report-issues,
a script that can be run on the command line (or daily via cron) and
displays a list of errors and warnings from the specified glob of hosts and
log files. By default it does all hosts for the current day, we've got it
running nightly so we can work through the issues each morning. It's worth
noting that sometimes in the output the same failure occurs more than
once. This is because puppet retries certain operations - such as retrieving
a resource.
There is actually a lot of useful information in the puppet reports. To start with I've added a todo item for a script that notes persistent errors (the same issues over two or three runs) that I'll hopefully get to this month. Maybe.
If you're running puppet in production you owe it to yourself to turn on reporting and set up some processes around it. While puppet makes it easy to perform action at a distance you still need to close the loop somehow.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 20:57 | /tools/commandline | Permanent link to this entry | This entry + same date
Who Watched the Watchmen? Me!
On Friday night I was as predictable as most of the people in my
feedreader and was camped down at 5PM for the evening showing of Watchmen - and I enjoyed
it.
It's been a good few years since I read the original graphic novel so I'm not as likely to pick out little errors and omissions (like the Silk Spectre looking for a lighter in the book and just being nosy in the film) but I thought the story was a very good, and close, adaptation. The fight scenes were well done and extremely short, sharp and brutal and the film utilised its 18 rating well.
I'm interested in how easy to follow the film will be to someone who's never been exposed to the source material. There were lots of little bits and pieces that might not fit together that easily. I thought the pacing was decent, I was never bored and didn't get too fidgety.
Now, spoilers
I thought the idea of using Dr Manhattans power as the basis for the destruction was an acceptable alternative to the novels ending. Trying to stick to the novel would have required a lot more leg work and slowed the pace. I was a little surprised at how most of the heros seemed to have some degree of super-strength but that's artistic license. As for the killing of Rorschach I can see why they did it that way and increased the drama but I didn't like it. The original handled it much better and without all the over the top responses.
Although I missed the giant squid I'll still give it 8/10 for fans of the novel, 6/10 for others.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/09 20:28 | /movies | Permanent link to this entry | This entry + same date
Wed, 04 Mar 2009
First Steps in Github
I finally decided to set up and start using a github account and my early
impressions are that it's quite slick and very userfriendly.
Apart from an annoyance where I couldn't see my pushes for a little while (I think I fell afoul of some caching) setting up an account and adding new repos was simple. Pushing from my actual dev machine just worked and I've now been bitten by the github bug. I feel the need to push code...
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/04 22:22 | /tools/online | Permanent link to this entry | This entry + same date
Tue, 03 Mar 2009
Personal Git Milestone - First Accepted Patch
It's been a day for nice little technical surprises. On the tube ride to
work this morning I started flicking through Cisco Routers for the
desperate (2nd edition) and found a quote on the first page from the
1st edition book review I did a couple of years ago.
I also had my first fully git workflow patch accepted by upstream. It was only a couple of lines of code but it means I'm gradually getting comfortable with the git toolchain.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/03 22:12 | /misctech | Permanent link to this entry | This entry + same date
Mon, 02 Mar 2009
Mozilla Ubiquity - Puppet Types
I like Ubiquity. It
puts a lot of the sites I used on a regular basis close to hand without
making me dig through my bookmarks (or del.icio.us account). In a
small burst of productivity, and to avoid real work, I decided to put a
command together for the Puppet Type docs at Reductive Labs.
If you have the Ubiquity plugin installed you should be able to install a copy of the command from the Ubiquity Puppet Types Command page. Writing a simple command like this one was surprisingly easy (especially using command-editor) although I don't (yet) know enough jQuery to add some of the other features I want.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/02 22:55 | /tools/firefox | Permanent link to this entry | This entry + same date
Sun, 01 Mar 2009
London PM Moose Talks - Feb 2009
I wasn't able to get to the actual talks but luckily the Moose talk slides are now all
online (apart from Moose for Ruby programmers which has instead been
expanded in to a blog post). By all reports it was another excellent night
and I'll have to keep the evening free for the next one.
Now I've read the slides and heard so much positive feedback I think it's time I tried Moose for a couple of projects. The initial install is still a little daunting due to the sheer number of modules it pulls in (although my main dev machine has most of CPAN installed anyway) but the benefits it seems to bring (I especially like the look of MooseX::Getopt) may be well worth the trade off.
If you're not familiar with Moose have a look at the slides, it's great to see a lot of the Perl6 promise here to use right now.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/03/01 23:03 | /events | Permanent link to this entry | This entry + same date
Sat, 21 Feb 2009
GDB Pocket Reference - (Very) Short Review
If you already know GDB then this book might
be useful. It's full of command summaries and option listings but
lacks an actual introduction or any walk through examples.
A google for GDB tutorials bought back some well written intros with actual sample code I could work through which is probably a more useful approach for most people.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/21 12:05 | /books | Permanent link to this entry | This entry + same date
Wed, 04 Feb 2009
Nagios check_http flaps
We recently had an odd one where the Nagios check_http check, which was
both checking for the presence of a string in the response and that the page
loaded in a certain time frame, went from reporting a 'CRITICAL - string
not found' to a 'HTTP WARNING: HTTP/1.1 200 OK'. My first thought, as
this was a site pending migration, was that the URL had moved to a
slower machine with the fixes released to it. Alas, it's seldom that
obvious.
It turns out that somewhere in the Nagios check a slow page that exceeds the -w options threshold overrides the fact that the string is missing, even though that's a warn replacing a crit. Bah.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/04 16:53 | /sysadmin | Permanent link to this entry | This entry + same date
Tue, 03 Feb 2009
Simple, Single Document Bookmarks in vim
I like vim, I think it's a great editor
worth investing time and effort in to learning but I also think it's one
of the most horrible things to watch an inexperienced user typo his way
through while you're urgently waiting for them to finish the damn edit.
My favourite one this week (and it's only Tuesday) is looking for
probably unique phrases that you can later search for to return to a
specific part of a document.
In an attempt to stop my laptop getting any more back of the head shaped
dents in it from when I've failed to restrain myself I thought I should
point out a much simpler way of doing this. Once you're at the part of a
document you want to return to press m<letter>. This
sets a mark. To return to it press '<letter>. That's it.
No more pasting in chunks of a string hoping it only occurs once in the
damn document. If you need to mark a couple of locations then fine just use
different letters to set and return to the places you want. And save me
sending another laptop back in for warranty.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/03 22:05 | /tools/commandline | Permanent link to this entry | This entry + same date
Splitting Syslogs by Facility
Logs are a wonderful thing. If done correctly they point out the source of
all errors, show you what's running slow and contain useful information on
how your system is running. At every place I've ever worked they've been
busy, full of odd one offs and too often overlooked.
I'm going to be doing a fair bit of log processing next week so expect lots of little toolchain scripts like syslog-splitter.pl to be checked in to git and mentioned here.
syslog-splitter takes a logfile as an argument and breaks the logfiles
in to many smaller units, one file per facility (which contains all the
lines for that facility from the logfile), to make it easier to process. I
seem to invoke it followed by wc -l out/* | sort -nr when on
new machines to work out where I need to invest some time. Over the next
week or so I'll come back to the topic and show how I'm reducing the noise
to help me find the important lines.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/02/03 21:49 | /sysadmin | Permanent link to this entry | This entry + same date
Fri, 30 Jan 2009
MySQL Uni - Scalability Challenges in an InnoDB-based Replication Environment
I recently 'attended' my first MySQL University presentation - Scalability Challenges in an InnoDB-based Replication Environment.
The service itself is great, you sign up, log in and then
watch the speaker present in one window while listening to him speak and
reading the slides (in the main part of the screen). Everything you'd expect
really.
The subject wasn't anymore exciting than you'd guess (but what do you expect with that title?) but the speaker knew his stuff and a couple of the Solaris commands shown will be useful to me in the future. Most valuable for me from the time invested in watching is that between this and the Jan 09 LOSUG I'm being pushed to invest some time in OpenSolaris and potentially deploy an instance or two of it in our staging environment for performance and debugging reasons. While Linux is pushing the boundaries in many ways Solaris is still ahead when it comes to working out the why, when and what.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/30 21:19 | /presentations | Permanent link to this entry | This entry + same date
LOSUG - Jan 09 Wrap Up
I'm going to try and get to more LOSUG meetings this year and the January presentation by MC
Brown has done nothing to put me off. Although some of the audience tried
their best...
First up - the good. The actual presentation, MySQL/DTrace and Memcached, was very well done. The speaker was funny, well rehearsed and knew his material extremely well. The MySQL DTrace probes are made to be used in demos and are very enticing.
Now for the bad. The talks technical coverage was quite light on the MySQL/DTrace part and the memcache section was painful. Big parts of the audience just didn't seem to get the idea. The speaker had to constantly backtrack to keep more than a handful of us with him. I had to resist the urge to join in and ask questions like 'how do you do joins?' just to see how big the vein on his forehead would get.
Still, the speaker must have been good as I've already played with the OpenSolaris Live CD they gave away. I'm looking forward to next month.
Like this post? - Digg Me! | Add to del.icio.us! | reddit this!
Posted: 2009/01/30 21:17 | /events | Permanent link to this entry | This entry + same date
Copyright © 2000-2005 Dean Wilson
