Never Fails...

Minding my own business yesterday when work called. Someone here was needing several items restored from backup tapes and there was someone already on-site to handle the task. No problem. Except we have changed tape drives since 2004...from the 40GB DLT tape drive to a much more robust Ultrium tape drive. So, I needed to come into the office to put the old tape drive back in place so we could perform the restore. Should be pretty simple...swap out the tape drives and be on my way. But, as Lee Corso says, not so fast my friend.

We took down the server and I went to work swapping out the drives...thinking 5 minutes should take care of my duties. Once the tape drives were changed, I turned back on the server. The file server would need to be up quickly because there were people waiting to do critical work (thus, the need for the restore on a Sunday.) It recognized the "new" tape drive...success. Well, maybe not. Soon I received two errors - PXE-E61 and PXE-M01. I had a good idea what they meant...and then I received the devastating "Invalid System Disk" error message. Hmmmm...all I did was change a tape drive? Not a hard drive...I never even cracked the case on the server itself.

So, what could have happened? Immediately I went into the BIOS and saw that HDD was not an option on boot devices. Not good. Apparently the reboot revealed a problem with our server. We called HP support and after a BIOS upgrade and a little troubleshooting, soon learned that our Smart Array Controller had gone bad. Hooray for that...

Luckily we had another server here that was less vital than our file server and I was able to steal the controller from it and get it into the server and that fixed our issues. We were able to get the old tape drive cooking and everything went smoothly from there. But a 5-minute issue turned into a 2-hour ordeal pretty quickly. I waited around while we did the restores to make sure everything was in working condition.

Needless to say, we were lucky that we had a non-essential server here the same model as what went bad yesterday...otherwise we would have had some very upset folks here. We have a 4-hour response time contract with HP but 4 hours would have been too long. I ended up in the office from 2 p.m. to around 8 p.m. last night...not horrible I guess, all things considered. So far this morning, no problems. Swapping the tape drives back at lunch today...hopefully that will go smoothly too...if not, I'm going home.