On a cold winter morning at 5:30AM, the Appsonnet team starts getting umpteen alerts that servers, services, Internet are down. Talk about needing a wake-up call and getting exactly that. Its a client site where apparently everything has stopped working. Hmm Power outage, crosses my mind.
I try logging in remotely to find the cause but am unable to access the client’s system, an obvious fact since everything is flashing red on the NiTO dashboard. Now comes the feeling where you know youre in for a long Friday! Little did I know- the long day is soon going to turn into a busy weekend. When your client is in the advertising industry, its a given theyll have deadlines to meet, particularly when Christmas is quickly approaching.
Next step -- call the client, and plan to go onsite while communicating with anyone onsite to assist in a visual rundown of the ISP switch and firewall.
Luckily, the on-site security guard is ready to help. The lights are green, comes the answer, which doesnt do much unless you count the fleeting moment of triumph when we think the Internet is up and something else is wrong. So battling traffic at 6:30am, man the city is getting busy, were onsite within 90 minutes . The lights are not green ð BUT everything else is good, its just the Internet and maybe the firewall.
So we call the ISP and wait on hold…. I mean at 7:00AM how long is their queue?
Once the ticket is created, we have the permission to reach out to their Meraki team to troubleshoot the switches and firewall. Lo and behold, the firewall is defective. We call the office to ship us a loaner firewall, but shipping would take most of the day traffic battle again put in our loaner and wait to get a replacement from the ISP. Client happy, Appsonnet happy too!
NEXT DAY -- Saturday two of us show up to replace the firewall but yesterdays nice security guard about-faced and now we dont have access so we wait…
If you work or live in downtown, you know one doesnt actively go out of ones way to kill time, because a lot of times it finds you. Now picture a shy, reserved and handsome young man = Mike = my partner-in-crime sitting outside, killing time in the middle of a busy city. Suddenly, clearly taken aback by his rather good looks, someone approaches him and well errr lets just say strikes up an uncomfortable conversation! Now I dont know what Mike is thinking, but for me, killing time becomes rather entertaining as I watch Mike turn flush-red in an instant while attempting to sort out whats going on.
I guess what they say is true -- when youre having fun, time seems to fly by, because before we know it, ACCESS GRANTED! Hoping to be in and out fast the next 10 hours is basically following orders from the ISP to try-this-and-try-that -- their tech team wouldnt give us access to their devices which is usually reasonable. But we are onsite and have first-hand view and the experience to know what is wrong, so not listening to us very unreasonable. Finally after escalating more than a couple of times, at 2:00am in the morning, were given access to the devices and within minutes systems are up and running.
Did the weekend pretty much fly by helping our client? Sure, and job well done! Did we learn an important lesson? Of course -- never wait outside alone. ð
Well ok, maybe we learned two lessons. When one tech isnt listening to reason, keep at it and eventually theyll listen. ð