Thursday, Dec 2nd 2010
Outage on Dec 1st 2010
We’ve been growing really fast, thanks to our amazing users that come to the site everyday to get their daily dose of inspiration, and share their favorite images from the web using the site.
For that reason we need to constantly upgrade our servers. Having better servers and more servers means that the site can run better and faster for everyone. Just so you get the idea: we have 8 million page views every day, that’s 93 access every second. Can you imagine 93 people passing through the same door at the same time?
So, yesterday (December 1st) we moved our main server to a new hosting company, one of the biggest in the world, in order to have a high level service. To move the data we had to freeze and backup everything, then copy it to the new server, that’s why we were out of service yesterday morning. Which is a great reason to be 2 hours offline. And that’s how long it took, we came back and everything was really fast, many people were happy.
But this morning (December 2nd), something happened. Something really bad. Some crazy inexplicable error affected the main server hard drives, both of them (we have two in case one stops working) and everything was lost.
It took some time to figure out the extent of the problem, we were talking to the hosting company all day trying to fix it. The solution was to reset the server, make it work, and then copy the backup again.
So we lost some data. Basically everything from yesterday. We’re really sorry about that. I mean really sorry. In theory this should never happen, we have backups, and mirror hard drives, and a few other things to prevent that. But it all failed at the same time, that was definitely a lesson for us, we’re never 100% secure.
These things happen, it’s awful. Just like having to print your homework and the printer doesn’t want to work even though everything is fine. We’re still trying to understand what exactly happened, so we can make sure it doesn’t happen again, ever.
Thank you for your patience.
Fabio Giolito & We Heart It Team
Comments (View)