119 replies to this topic

[sidestand]

I'll ring Gary.........tell him to get his finger out & get it sorted

[Radio_am]

It's a big problem I'm afraid:

"This evening at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost.

We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department.

This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock.

We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.

There is no impact in any of our other five data centers.

I am sorry that this accident has occurred and I apologize for the impact."

This is from the people that host a major DNS, so it affects quite a few customers.

My hosts have set up replica DNS servers so we'll either have to wait until the network converges with the new DNS data details or the server host (them lot above) sort out their power issues.

That should be in less that 4 hours but who knows really. Sorry but there's nothing I can do and the hoster has done all it can until the replica DNS's get advertised. Hopefully it'll be a few hours

[sidestand]

I bet it's all down to Waspy, pissing in that French phone box

as it's stopped raining here, I think I'll bugger off out for a ride until Gary's sorted all his replica wotsits

Edited by sidestand, 01 June 2008 - 04:10 pm.

[Radio_am]

I blamed Waspt too, the French Flic's are after him apparently. They've DNA'd his piss and are looking for a bloke with a high proportion of butter on his skin.

I've set up a twitter site that will have updates (as well as here of course) and people can see that at: http://twitter.com/gocmc

[sidestand]

I suppose we ought to apologise in advance to Guiness, as the tone will get lowered on Carpe pretty quickly if all the CMC lot don't have anywhere else to go for a while

[sidestand]

Had a quick look, but its all double dutch to me
Ibet we haven't paid the bill & they've cut us off

[Radio_am]

Nah, what happened was a datacentre in the United States of America had a power problem that resulted in an explosion and the loss of three walls.

It also meant that the DNS servers (they are the bits that tell you where to go when you type www.gocmc.org) that 3dpixelnet use have gone 'poof' and so the internet doesn't know where CMC is.

When the new DNS set up by 3dpixelnet take over or the walls are put back in Houston and the servers can be powered up again everything should be back to normal. It's caused a bit of a stink because these DNS's should really be clustered over several servers and locations...

Anyway, that's where we are... Or perhaps aren't

[Radio_am]

LOL...

You never know.

[Radio_am]

Plans are for the DNS's to be back for midnight UK time.

We live in hope

[sidestand]

That's long past my bedtime Gaz - I'll try it out tomorrow

BTW - D'you start yer new job tomorrow, or are you tossing it off for a while

[David H]

Nah, what happened was a datacentre in the United States of America had a power problem that resulted in an explosion and the loss of three walls.

Can they not just put a tent up.

[Yellowasp]

mine out too

[David H]

Well still no CMC.

Gary, your twitter has a sad face. Does that make it a bitter twitter?

[Petrust]

Bet it's Gary's fault really - he told everyone he was going out on the pi$$ friday night

[Radio_am]

It's still not working even though 3dpixelnet have sorted out alternate nameservers. I think it'll just have to wait until the IP's get updated around the internet.

Dave... As the bloody thing still isn't working it's a bit of a Shitter Bitter Twitter

Oh.. Pete... I did have a bit of a big on Friday night and still had the hangover to prove it yesterday buit I swear I didn't break anything let alone fly to Houston and blow up their datacantre.

Edited by Radio_am, 02 June 2008 - 09:44 am.

[tdm850rider]

Have you been using bookmark links? Try typing in the URL fresh.

[Radio_am]

Nah Tam, it's fooked for me but some people are getting on. I don't think it's propogated around the internet fully yet

I'd hate to think what they are writing about me!