The great Heart Internet data center meltdown

by Tim Hill (aka Doodled)

February 13th 2016
The Heart Internet data center meltdown
The Heart Internet data center meltdown was bad, their crisis management made it worse.

Website hosting isn't my specialty, SEO and online marketing are. The great thing was I didn't have to be ... until Heart Internet went pear-shaped. If any of my client's rankings were suffering due to their host (slow response, lots of downtime, etc.) the answer was always simple - move them to Webfusion. For larger sites move them to a Webfusion VPS.

I've used Webfusion VPSs since the 10th of whenever and it was one of those services you didn't have to think about - it did exactly what it said on the tin. For the occassional headache you could pick up the phone and have issues resolved in real time.

I hosted clients on this UK based server wherever they were - Australia, the US, Europe - and it just worked.

Unfortunately last year Webfusion discontinued their VPS service and when the time came to upgrade they recommended Heart Internet, a sister company. I moved a few clients over although I was never truly happy about the setup. Plesk Power Panel was suddenly a paid extra and there was no telephone support, not even live chat, just a ticket system leaving you with no idea when a response would come.

But technically the hardware worked well so I rolled with it ... until one of Heart Internet's data centers collapsed ... or as it was reported by The Register - Heart Internet in 22-hour TITSUP after data centre power stuffup. I should point out at the time of writing this we are more than 48 hours in.

A badly managed crisis.

This type of melt down just should not be possible in one of the UKs largest hosting company so it points to corner cutting, perhaps in both infrastructure and staff. It tells you that Heart is no longer a good place to host websites. Even when the current outages are recovered unless they are clear on how they are going to change the next disaster will just lurk in the shadows.

But it was how the crises was managed that made a bad situation worse.

It was quickly obvious that Heart had no proceedure in place for what to do in a crisis. Obviously their support teams became overwhelmed but the problem was that they stayed overwhelmed. At the time of writing I have a ticket that has not been responded to in 22 hours!

This left customers dangling but then they made matters worse. They carried out mass ticket closures to flush their bogged down system and left messages such as "please re-open this ticket if your website is still down". So already angry customers logging in to check what was going on became even more infuriated to find their tickets closed.

However the largest sin of all was that they did not keep their system status page up to date. As I write now it has not been updated for 21 hours. This causes the obvious domino effect - no one knows what is going on so they all raise tickets which then overwhelm the support staff .... again.

Finally they did not, or were unable to, provide sensible error messages for the websites which were down. A simple "This site is experiencing technical difficulties, we are currently working on restoring full service and apolgise for the inconvenience" would have done.

Smart Hosting, a competitor to Heart, posted on their Facebook page "Due to outages at a competing provider, we are currently experiencing extremely high volumes of sales enquiries." as the inevitable exodus took hold.

Its not always the crisis, its how it is managed

I doubt Heart Internet will be able to overcome this, they may have to rebrand to reduce the fall out. However I hope they remember that the data center failure was only part of the crisis, the management that followed should be remembered as the number one reason they lost so many clients.

Update: Their status page has now been updated. It is Saturday afternoon and they have announced that normal service should be resumed late Monday. If they make it happen this will mean many websites will have been down for 5 days - unheard of for a hosting company in this day and age.