Get your own
 diary at DiaryLand.com! contact me older entries newest entry

7:16 p.m. - 2002-09-16
late nights just seem to know when to happen lately
Okay, so yeah... I was all ready to leave and junk around 3PM and suddenly the lights flicker a bit and we hear a crash of thunder close by. No one thinks anything of it for a minute and then suddenly none of the servers are responding, we have no phones and life has gone to shit in one split second.

Seemingly, lightning struck some electrical junction or switch or something really near by and consequently surged, blew out our UPS system on the first floor which caused a power fluctuation in the entire building and caused every last server in the building (except two of three partitions in the Superdome) to come crashing down. Isn't that just fun?

And wouldn't you know it, my production server is the ONE partition on the superdome that went down. JOY!

So we all run around like army soldiers, being banished to the cold room where those servers are kept just in case we need to access them from the console to make sure everything still exists... I, luckily, was sent back to my desk within a half hour and waited another 40 minutes while the machine was coming back up to start hacking away at the things that absolutely NEEDED to be brought back up and checked out. Out of 8 of the production databases effected by the hard crash, I only lost 2. One of those came right back after the restore. Then there's my friend CSSTOREH who has decided to be a royal pain in my ass and has seemingly deleted it's log files from prior to the backup last night, therefore rending the back up USELESS! Ain't this just a fun time waiting to happen?

So I harass the hell out of my unix guy, searching for missing log files and consequently just having him restore a two week old backup to try to get this sucker back up. Good lord what a mess. But I keep reminding myself that this could have been a whole lot worse. I could have lost our main database, which albeit is thankfully replicated unlike this PIMA.

And it seems a weird thing to be saying indeed but I should be home doing my homework for my class tomorrow. I -want- to go home and do homework? What the hell is THAT all about?

Thankfully though, He is not home waiting for me or I'd be pining to go home more than I already am. I'm hungry but oddly enough I had packed an extra pop tart this morning so I ate it around 4... it's 7:30PM now and I'm STILL trying to get this damn thing back again!

Ahh well... I can't even be on Yahoo or AIM because our proxy server got FRIED in the surge. A strange thing since there's a unit just outside our building that's supposed to suppress all of that. I guess it must have been struck directly... no idea, all's I know is that the electrician said there was smoke rising from the UPS unit just after the charge hehe... what a disturbing but amusing picture that is...

Back to work for me, hopefully this time the restore works... I want to go home.

*hugs to all*

 

previous - next

 

about me - read my profile! read other Diar
yLand diaries! recommend my diary to a friend! Get
 your own fun + free diary at DiaryLand.com!