How Tim Burton and WP Super Cache brought my sites to their knees

We felt like Alice tumbling down the rabbit hole last night. News broke that the the trailer for Tim Burton’s Alice in Wonderland had leaked onto the internet, Sara nabbed a copy, threw it up on Daily Shite and then all hell broke lose.

I didn’t think much of it at first, but then the traffic started flowing in and things started to go haywire.

All of our sites are hosted with DreamHost on various accounts, but O’Flaherty, Suburban Oblivion and Daily Shite are all hosted under the same account, which means they’re all residing on the same server (along with hundreds of other customers sites I would presume), so when my site started throwing 500 errors as Daily Shite came under heavy traffic I knew I’d done something wrong.

The worst thing about being on a hosted server is that it only takes one of your sites to be poorly configured in order to go over your resource allotment and take the others down as well.

So as the traffic spiked on Daily Shite I was left confused as to what was happening. The sites had been working fine under reasonable load, I had WP Super Cache installed so what could it be?

To make matters worse, getting into the admin panels of the sites to change settings was proving difficult as they all kept throwing “Internal Server Errors”.

I’ll cut a long story short and get to the chase, after a quick SSH login it became evident that the reason the sites were throwing wobblies was the very plugin that was installed to help the sites deal with heavy loads – WP Super Cache.

In my eagerness to get everything running smoothly and quickly, I’d enabled Gzip compression on all of the sites. Bad move on my part.

While Super Cache was doing what it should and serving only cached versions of the pages, instead of the whole batch of database calls it’s supposed to prevent by not recreating the page for every user, it was now using CPU cycles to compress the data being sent to readers browsers.

Now this is fine on an average load day, the extra CPU load from compression is negligible and provides a minor speed increase in load times for anyone viewing the site.

But when you’re seeing a spike like we did yesterday, it just brings the entire system to its knees.

Once I had Gzip compression, as well as “Coarse File locking” disabled everything came back to life and ran like a dream throughout the entire 2-3 hour traffic spike.

So, if you’re expecting or hoping for a huge traffic load on your site and are on a hosted server here’s my recommended settings for WP Super Cache:

WP Super Cache config

  • Don’t cache logged in users – Disabled (no check mark) If a logged in users visits the site with this enabled they force a rebuilding of the cache which uses up valuable resources under a heavy load.
  • Clear all cache files when a post or page is published – Disabled (no check mark) This can significantly increase posting time as a the cache is cleaned out.
  • Cache Rebuild – Enabled (checked box) – Serve a supercache file to anonymous users while a new file is being generated. Recommended for very busy websites with lots of comments. Makes “directly cached pages” and “Lockdown mode” obsolete.
  • Coarse file locking – Disabled (no check mark) You probably don’t need this but it may help if your server is underpowered. It can cause some server configurations to lock up so best to leave disabled.
  • Mobile device support – Enabled (checked box)
  • Super Cache Compression – Disabled

One final tip for working with WP Super Cache: Don’t forget your .htaccess files. When you set up, or upgrade the plugin you have to insert (and check for changes in) rules in not just your root .htaccess file but you must also create a .htaccess file in “wp-content/cache” and manually populate it with the rules given in the plugins admin page. Also, don’t forget that adding mobile support requires and additional set of rules to be added manually.

Forgetting to add the rules can lead to trouble and the plugin will not work as it should.

Now you should be prepared to be dugg or any other massive traffic spike and won’t be caught with your trousers down like I was :)


  1. says

    wow, supercache could really be the answer. I'm expecting at least 400k pageviews in 4 hrs and probably 50-60K unique visits. Do you think super cache can handle that?

Leave a Reply