I’m sure that, by now, many of you noticed that things were a bit “off” yesterday. Pages would load funny, or you’d be greeted with a strange white screen with the simple word “error.”
There’s a good reason for that.
At about 1:30PM Eastern, our server crashed, and crashed hard. Thanks to CloudFlare and the an incredibly aggressive caching strategy, though, we were spared from a lot of the horrors that could have happen, including (but not limited to) total downtime. Thanks to the good folks at Digital Ocean, we were able to get the site up back and running with a decently new backup. By “decently new,” I mean about “two weeks old.” So, last night, we worked tirelessly to get things back up, running, and simply fabulous for you guys.
We’ve conducted a root cause analysis, and found that our WordPress install was just borked by one of our security modules. The plugin caused an unauthorized transfer of permissions, which pretty much crippled the site and the server, due to the generally restrictive conditions. To keep this from happening again, we’ve replaced this module, and now have a daily backup running. Rest assured, I do not want this to happen ever again.
So, anyway, this was our “school of hard knocks” moment. It shouldn’t have happened, and we’re planning so it doesn’t in the future. I apologize for the inconvenience yesterday, everyone. We’ll be back to our normal routine starting today!