On a cold winter morning at 5:30AM, the Appsonnet team starts getting umpteen alerts that servers, services, Internet are down. Talk about needing a wake-up call and getting exactly that. It’s a client site where apparently everything has stopped working. “Hmm… Power outage,” crosses my mind.
I try logging in remotely to find the cause but am unable to access the client's system, an obvious fact since everything is flashing red on the NiTO dashboard. Now comes the feeling where you know you’re in for a long Friday! Little did I know- the long day is soon going to turn into a busy weekend. When your client is in the advertising industry, it’s a given they’ll have deadlines to meet, particularly when Christmas is quickly approaching.
Next step - call the client, and plan to go onsite while communicating with anyone onsite to assist in a visual rundown of the ISP switch and firewall.
Luckily, the on-site security guard is ready to help. “The lights are green,” comes the answer, which doesn’t do much unless you count the fleeting moment of triumph when we think the Internet is up and something else is wrong. So battling traffic at 6:30am, “man the city is getting busy,” we’re onsite within 90 minutes…. The lights are not green :( BUT everything else is good, it’s just the Internet and maybe the firewall.
So we call the ISP and wait on hold.... I mean at 7:00AM how long is their queue?
Once the ticket is created, we have the permission to reach out to their Meraki team to troubleshoot the switches and firewall. Lo and behold, the firewall is defective. We call the office to ship us a loaner firewall, but shipping would take most of the day… traffic battle again… put in our loaner and wait to get a replacement from the ISP. Client happy, Appsonnet happy too!
NEXT DAY - Saturday – two of us show up to replace the firewall but yesterday’s nice security guard about-faced and now we don’t have access…so we wait...
If you work or live in downtown, you know one doesn’t actively go out of one’s way to kill time, because a lot of times it finds you. Now picture a shy, reserved and handsome young man = Mike = my partner-in-crime sitting outside, killing time in the middle of a busy city. Suddenly, clearly taken aback by his rather good looks, someone approaches him and well … errr… let’s just say strikes up an uncomfortable conversation! Now I don’t know what Mike is thinking, but for me, killing time becomes rather entertaining as I watch Mike turn flush-red in an instant while attempting to sort out what’s going on.
I guess what they say is true - when you’re having fun, time seems to fly by, because before we know it, ACCESS GRANTED! Hoping to be in and out fast… the next 10 hours is basically following orders from the ISP to try-this-and-try-that - their tech team wouldn’t give us access to their devices – which is usually reasonable. But we are onsite and have first-hand view and the experience to know what is wrong, so not listening to us – very unreasonable. Finally after escalating more than a couple of times, at 2:00am in the morning, we’re given access to the devices and within minutes systems are up and running.
Did the weekend pretty much fly by helping our client? Sure, and job well done! Did we learn an important lesson? Of course - never wait outside alone. :)
Well ok, maybe we learned two lessons. When one tech isn’t listening to reason, keep at it and eventually they’ll listen. :)