It was a chilly evening drive to the Scarborough Data Centre, a week into turning on monitoring for our new Client's systems, things started beeping red. We had identified redundancy issues with the firewall and in addition there was an odd set of alerts on a Power Supply of one of the load balanced servers. Upon arrival we opened the secured cabinets and faced the tangle of cables; the "fashion statement" the client had been living with. We looked at each other and said, "We have to fix this!"
The first step was to carefully push aside the "curtains" to get a view of the back of the cabinet. Just imagine 144 multi coloured Ethernet cables, one end into the switch draping straight down to the floor, and looping back up in the middle of the cabinet before connecting to its destination; it was the often seen "IT overload get things done NOW!" Syndrome.
We proceeded to track down the faulty power supply by endeavouring to carefully keep the curtains open, hoping that a power cable would not come undone. We test the power supply and confirm that the firmware is old, so the power gets reported as faulty. It's a redundant system, so we update the firmware, after which all seems well.
In our efforts we notice that there is an unusually large number of power plugs not being used and redundant power connected to the same power bar. There is also far too much space between racked servers in one cabinet while having nearly half a cabinet empty next to it.
We did some simple math, engaged the client in a meeting and explained that too much power was allocated for their needs and too much space was being wasted within the cabinets. They asked us to verify our math against their future needs, and take the necessary steps to implement, which resulted in savings costs in excess of $2,500 per month!.
Next step: assign a lucky team member to fix the case of the droopy curtains.
Bottom Line: Savings costs in excess of $2,500 per month! = A very Happy Client.