Rebuilding all of my websites

I’ve had quite busy recently – on Thursday last week I discovered all of my web sites were offline, which resulted in me moving to a new hosting provider, and rebuilding every bit of content. So how did I do it?

Going offline

I first realised something was wrong when I discovered all of my web sites displaying the following ominous error message:

 'Website Suspended' message from cPanel

I checked my email, and I couldn’t find any notification from my hosting provider that my account was suspended – a pretty shit job from them!

However, I wasn’t exactly surprised, as over the past few years I’ve been receiving these automated emails from their system:

Your hosting account with username: [XYZ] has over the last few days averaged CPU usage that is in excess of your account allocation.

This could be caused by a number of factors, but is most likely to be due to a misconfigured installation of a 3rd party script, or by having too many features, modules or plugins enabled on your web site.

If you simply have a very busy or popular web site, you may need to upgrade your account which will give you a higher CPU allocation. Please contact our support team if you need help with this.

Until your usage average drops back below your CPU quota, your account will be throttled by our CPU monitoring software. If your account continues to use more CPU than what it is entitled to, you risk having your account suspended.

All up I was running about a dozen different web sites from my single shared web hosting account, and over the years I’ve have had to increase the amount of resources available to my account to deal with the increasing load.

Eventually I ended up on a ‘5 site’ package from my hosting provider, which they were charge me almost $300 a year to provide – a steep price, but I was too lazy to move everything to a new web host, so I just kept on paying it.

Having all of my sites go offline was enough of a push for me to move somewhere new!

What needed to be moved

All up my online presence consisted of a dozen different sites spread across a handful of domain names, running a mix of open source code and code I had written myself. With my original web host inaccessible, I had to rebuild everything from backups.

You do have backups don’t you?

Welcome to the western suburbs

The rebuild

I had been intending to move my web sites to a virtual private server (VPS) for a while, and having to rebuild everything from scratch was the perfect excuse to do so.

I ended up deciding to go with Digital Ocean – they offer low-ish prices, servers in a number of different locations around the world, fast provisioning of new accounts, and an easy migration path to a faster server if you ever need it.

After signing up to their bottom end VPS (512 MB RAM and a single core) I was able to get cracking on the rebuild – they emailed me the root password a minute later and I was in!

As I had a bare server with nothing installed, a lot of housekeeping needed to be done before I could start restoring my sites:

  • Swapping over the DNS records for my domains to my new host,
  • Locking down access to the server,
  • Setting up a swap file,
  • Installing Apache, MySQL and PHP on the server,
  • Creating virtual directories on the server for each separate web site,
  • Creating user accounts and empty databases in MySQL

I’ve only ever played around with Linux a little, but after 30 minutes I had an empty page appearing for each of my domain names.

To get my content back online, thankfully I had the following backups available to me:

  • I run three blogs on the open source WordPress software, so I could just install that from scratch to get a bare website back
  • My main photo gallery on the open source ZenPhoto software, so that was another internet download
  • Each blog and photo gallery uses a custom theme, of which I had backups on my local machine to re-upload
  • I keep a mirror of my WordPress uploads on my local machine, so I just had to reupload those to make the images work again
  • When I upload new photos to my gallery, I keep a copy of the web resolution version on my local machine which I was unable to reupload
  • Every night I have a cron job automatically emailing me a backup copy of my WordPress and ZenPhoto databases to me, so my blog posts and photo captions were safe
  • Some of my custom web code is available on GitHub, so a simple git pull got those sites back online

Unfortunately I ran into a few issues when restoring my backups (doesn’t everyone…):

  • My WordPress backup was from the day before, and somebody has posted a new comment that day, so it was lost
  • I had last mirrored my WordPress uploads about a week before the crash, so I was missing a handful of images
  • The last few months of database backups for Rail Geelong were only 1kb in size – it appears the MySQL backup job on my old web host was defective
  • Of the 32,000 photos I once had online, around 2,000 files were missing from the mirror I maintained on my local machine, and the rest of them were in a folder hierarchy that didn’t match that of the database

I wasn’t able to recover the lost comment, but I was able to chase up the missing WordPress uploads from other sources, and thankfully in the case of Rail Geelong my lack of regular updates meant that I only lost a few typographical corrections.

As for the 2,000 missing web resolution images, I still had the original high resolution images available on my computer, so my solution was incredible convoluted:

  • Move all of the images from the mirror in a single folder
  • Use SQL to generate a batch file to create the required folder structure
  • Use more SQL to generate a second batch file, this time to move images into the correct place in the older structure
  • Run a diff between the images that exist, and those that do not
  • Track down the 2,000 missing images in my collection of high resolution images, and create a web resolution version in the required location

Three hours after I started, I had my first win.

Unfortunately I found a number of niggling issues throughout the night.

By 2am I was seven hours in, and had managed to get another domain back online.

Eventually I called it quits at 4am, as I waited for my lethargic ADSL connection to push an elephant up a drinking straw.

I spent the weekend out and about so didn’t get much time to work on rebuilding my content – it wasn’t until the fourth day after my sites went down that I started to track down the 2,000 missing images from my photo gallery.

Thankfully I got a lucky break – on Monday afternoon I somehow regained access to my old web host, so I was able to download all of my missing images, as well as export an up-to-date version of the Rail Geelong database.

After a lot more stuffing around with file permissions and monitoring of memory usage, by Tuesday night it seems that I had finally rebuilt everything and running somewhat reliably!

What’s next

Plenty of people online seem to rave about replacing the Apache web server and standard PHP stack with Nginx and PHP-FPM to increase performance – it’s something I’ll have to try out when I get the time. However for the moment, at least I am back online!

You can leave a response, or trackback from your own site.

12 Responses to “Rebuilding all of my websites”

  1. Kevin ohashi says:

    Highly recommend Nginx+FPM if you need high performance. However, most sites really don’t need it in my experience. You can get a lot of extra performance just from using something like WP Super Cache/W3TotalCache with your existing Apache. It may not be quite the same, but it’s a big step in the right direction.

    • Marcus Wong says:

      At my old web host I used the W3 Total Cache plugin to cut the load on my server, and I’m still using it now – with that I can easily handle a few hundred page views a day. I’ve just got to see how it handles my next spike of a few thousand views!

  2. Matt says:

    Marcus a similar thing happened to me
    where i mistook my main SQL as a empty SQL
    And i deleted it checked the website
    to find out that the database was deleted
    and i had no backups but the website had little
    content and was very new i made a new sql ,edited the
    database file and it was back i am now making
    backups and i am going fine

    Matt 🙂

    • Marcus Wong says:

      I’ve almost done the same thing before – with a dozen different databases setup in MySQL, I’ve had to be careful to given them all meaningful names so I don’t get them confused!

  3. mich says:

    You get more done in a day than I manage to get done in a year.

  4. Tim Chuma says:

    Well done with getting it all back up so quickly. Am fretting about all my photos now as I moved to a photo gallery management system rather than hard-coded galleries that were easy to upload but a pain to maintain especially deleting individual photos and doubly so if it is one from over five years ago and I have to hunt for it.

  5. […] my various web sites can be difficult at times, and my experience the other weekend was no different. My day started normally enough, as I logged […]

  6. Tim Chuma says:

    gallery3 photo gallery stopped being supported and I now have 28k of photos in it and captions. If I had to migrate it manually I wouldn’t get anything else done 🙁

  7. […] The first option is unreliable because it delays page load times, and you can’t count on people visiting a low traffic web site, so the second option is what I put in place when setting up my server. […]

Leave a Reply