Dean Wilson@UnixDaemon: ... It's not even close to relevant.

Tue, 17 Apr 2007

No one likes a whinger - The systems fight back
After my little whine I logged in to do my last checks for the evening to discover that one of our webservers had died due to a hard drive going bang, our production environment Nagios box had lost one of its network connections and a chunk of our SAN kit was complaining about power issues. Turns out that most of these were due to a power surge that killed a network switch and three of the racks power strips. On the very plus side no one outside of the systems team noticed. Resilience is a wonderful thing when you get it right.

Woke up this morning, checked the ~~Nagioses~~ Nagii and found out that one of our other products database servers had gone boom (my fellow sysadmins were fixing that one) and the fail over had mostly worked. No interesting logs, no hardware problems and a three hour gap in syslog (and only syslog) to help explain the outage.

What have I learned? That the production servers read my blog. And they hate me.

Like this post? - Digg Me! | Add to del.icio.us! | reddit this!

Posted: 2007/04/17 21:32 | /sysadmin | Permanent link to this entry | This entry + same date

October 2008	5
September 2008	4
August 2008	12
July 2008	9
April 2008	4
March 2008	1
February 2008	1
January 2008	15
August 2007	2
June 2007	9
May 2007	6
April 2007	8
March 2007	31
February 2007	3
January 2007	21
December 2006	1
November 2006	4
October 2006	6
September 2006	32
August 2006	17
July 2006	14
June 2006	9
May 2006	13
March 2006	11
February 2006	16
January 2006	11
December 2005	1
November 2005	6
October 2005	19
September 2005	25
August 2005	16
July 2005	16
June 2005	13
May 2005	2
April 2005	19
March 2005	31
February 2005	20
January 2005	31
December 2004	21
November 2004	30
October 2004	32
September 2004	18
August 2004	7
July 2004	14
June 2004	5