My fellow anime fans, good evening.
If you logged over the past few days, I’m sure that you’ve seen a bit of craziness unfolding. I’d like to take some time to clear the air on this.
On February 11, at about 10:50AM Eastern, Anime Herald was hit with a cyber-attack, taking the site down for roughly four hours. In our initial assessment, we assumed that this was a Distributed Denial of Service (DDOS) attack, due to the overloading of resources – RAM and CPU – to the point of crippling the site. The site was down for four hours as we attempted to return to good working order.
That said, once the site was back in working order, we had a chance to sit down and look at the details. While this was a Denial of Service attack, it wasn’t exactly distributed. We were able to trace the IP back to a single user from Germany. This user was using a security exploit in Google’s PageSpeed extension, which was patched recently. Through the loophole attacker was able to gain access and generally flood our channel with garbage requests, taking the site down and keeping it down. On PageSpeed’s Release notes, this was issue 674.
Unfortunately, the damage was done – we were unable to restore the site to a good working order, and made the difficult decision on Sunday to start from a new VPS (Virtual Private Server) instance. Google PageSpeed was not installed this time, removing that vector entirely.
At about 7:20PM Eastern, Anime Herald went down for maintenance. A new site was spun up, and we began the long, hard work of loading our backups.
Right now, I want to give a gigantic shout-out to the folks at VaultPress, who have been maintaining daily backups of Anime Herald for the past few years. They’ve been an amazing partner in our recovery, and helped to completely restore the site to working order in just a couple of hours. Every post, every file, every item was back in-place as if nothing had happened.
Anyway, last night, while we did go down again for a bit of time, this was a much happier circumstance. Though the Herald was back online and things were stable, a massive traffic spike from one of our news items that evening left the server reeling. We were unable to really keep up, and our database provider crashed – causing temporary outages.
At about 8:50PM Eastern, an executive decision was made – we were going to upsize the VPS, doubling our RAM and storage capacity. The rollout began at 9:45PM, and the new server setup went live about five minutes later. We’re now stable, and should be well insulated from further traffic shocks.
I’d like to thank you all, for your patience in the time it took to get back on our feet. Also, I’d like to thank you for driving enough traffic to bring our previous server to its knees. That’s not sarcasm! I’m just so glad that you all put such trust into our work, and our organization.
In short, you all rock. Thanks for sticking it out when we had technical issues, and look forward to more amazing content from our kick-ass team.