12:00PM
It was a quiet day at the office that Friday, things were running smoothly, all servers humming along nicely, we love it! Heading out for lunch…
Ring…Ring… the mobile rang, it was our client, Peter, sounding frantic, sitting in a car travelling in Europe, hopping from city to city, meeting to meeting, “Our websites are down -- no one can get to any of them from anywhere in the world”, “The developers can login, but can’t hit the sites”. This was urgent, lunch would need to wait…
“We’re on it,” was our response, and moments later we identified the root cause: their ISP was closing their doors without notice, the external DNS was down and soon everything else would be too.
Despite our pleas, our attempt to buy more time with the closing ISP was hopeless. Peter was upset, frustrated and the order came, “Get us outta’ there now!”
1:00PM
We rented a van and rushed to the closing ISP site, took notes, and began to dismantle the equipment. Three cabinets full of hefty servers, one large storage SAN, and relocation options complicated by the 208V-only servers.
Peter’s team had a quarter cabinet in a second Data Centre, but it was much smaller than what we needed. We got in touch with the second Data Centre team; it was easy dealing with them. They could provide one full cabinet by 6pm with the power we needed. “Okay, lets boil this down to critical servers and the SAN. We can pull this off with one cabinet.”
5:00PM
The ISP’s doors were closing behind our back as we tugged the last power cord into the van. The sun was going down, lunch long missed, stomachs growling, “No time for food, we need to get to the Data Centre!”
6:00PM
We were at the second DC and began unloading the equipment to its new home.
7:00PM
As we started to rack-mount the equipment we noticed that the rack’s power was 120V…not the 208V we requested…”Uh Oh”… We called the second DC team again, they were trying to help, but couldn’t do the 208V in short notice. “Quick, time to get creative.”
8:00PM
We knew the client had an existing quarter cabinet with 208V. We could extend that power to the full cabinet, but the cabinets were too far apart. The one time we need an extension cable, it’s not to be found, let alone the special one we need.
Then we noticed that three quarter cabinets above the client’s existing quarter cabinet were empty. Feeling a bit lucky, we asked the second DC team whether we could use the existing cabinets temporarily while they get the 208V for the full cabinet next week? We get a “yes” and call Peter for approval to further trim down to the most critical servers to fit in the quarter cabinets.
11:00PM
Everything critical was almost racked in three quarter cabinets, a bit tight, but functional. We asked for vents to be put in place for better cooling. We proceeded to connect everything; double checked the connections, and started powering up…
12:00AM
First, power to the SAN: SAN won’t turn on…all lights are red. Unplug it, triple check tray cables…carefully a fourth time…two separate people check…something is not adding up.
1:00AM
We called the SAN’s vendor support line and were told, “Your client’s support contract has expired…” This was not right, we knew that the support contract was renewed just a few months ago. While we waited on hold, SAN support tried to contact someone from their accounting department, but at that late hour nobody was answering.
2:00AM
We call a guy in the local vendor support team who had helped us in the past. We’re given the cell phone number to an on call manager who can approve the work. We call him and he identifies a discrepancy in their payment records. We’re getting somewhere; he approves the request for support…lucky he never asks how we got his number.
3:00AM
We get a call back from the Vendor Support Call Centre with our magical answer, “…a secret button that auto flips and prevents power related disasters”. We flip it, turn on the system, lights start blinking and slowly turning green. It’s alive!
4:00AM
We called Peter for verification from him and his team that all was okay. They ran their tests…we grabbed some pancakes at the 24-Hour Breakfast across the street.
6:00AM
Got the “all okay”. We packed up and headed home for some shut eye.
8:00AM
Sun rising, Ring…Ring… the mobile rang, it was Peter, sounding relieved, “THANKS!!”
Bottom Line: the “Thanks” was worth it.
