If you don’t already know it, Dyn houses its servers and piles of high speed networking gear in top of the line colocation facilities across the world. This permits our servers to have access to redundant power (which they so dearly need) and redundant cooling (which they also need, or they tend to malfunction). The colocation also provides us with a building in which to connect to other networks, so we can get connected to the Internet.
One other service that a colocation facility provides is on-site hardware maintenance and troubleshooting. The service is really the generic catch-all service for all things you might want from your colocation. The service can install new equipment, cable up servers, move switch ports around, label your equipment, receive or send shipment, and debug problems as your remote hands and eyes.
This morning, we had a problem with some of our DynECT Platform servers. Not a major problem, because anycast re-routed DNS queries around the outage.
So – on to the hilarity – the technical support call went something like this:
Me: “OK, so we’ve had a problem with a power distribution unit, I’d like all the servers that are currently plugged into this unit moved to another.”
Engineer: “OK, doing that now.”
Yep, the engineer did that. As soon as he moved the power cords over, the affected servers came back online. But I wanted something else – it went like this:
Me: “Great, I see that those servers are online now. Could you please move some ethernet cables between two machines for me?”
Engineer: “Oh, I can’t do that, I’m an Engineer.”
Me: “Ummm, why not? Don’t you know how ethernet is wired up?”
Engineer: “No, sorry, we’re not trained on that.”
Me: “Ummm, so what do I have to do to get ethernet cabled moved?”
Engineer: “You need to open a ticket.”
Me: “But I opened a ticket for this already, the ticket says to handle power and ethernet moves and changes.”
Engineer: “Oh, they didn’t explain it to you? Engineers handle power issues, and technicians can do everything else.”
Me: “What a wonderful delineation between the two roles. Thank you.”
*#$%! – Seriously? What are they smoking? Engineers do power? Technicians do everything else? Silly, I say. One would think that a “higher-level” engineer should be able to perform all of the tasks that a “lower-level” technician should be able to do.
Anyways, in the end, we got it all sorted out and all systems are online again. This is thanks to keeping hot-spare equipment in all of our facilities for when hardware goes bad.
So that’s my fun colocation story – I want to hear yours! Post your funny colocation story in the comments below so you can blow off some steam too!