My fellow anime fans, good evening.
As you may have noticed, The Herald has been pretty much out of commission for the entire day. I wanted to take some time, in the interest of full disclosure, to explain the series of events:
At about 4:30AM Eastern Time (1:30AM Pacific), the core site was taken down in a cyber-attack. A massive spike in network traffic forced the database offline, rendering the site unusable. At about 8:30AM, we were able to bring things up again for a short period. Unfortunately, this didn’t last long, as we were hit a second time, rendering our site unusable again. Moreover, someone found a way into the server proper, as several core files for our nginx install were modified and vandalized.
In plain English, shit was fucked. Bad. A selection from the server config… the least profane portion… is below.
The Immediate Aftermath
The first thing we did was verify that we had active backups.
Thankfully, we did! Our last backup was taken two days ago – we had missed yesterday’s due to an odd server spike in the morning, and today’s due to the situation at hand. Unfortunately, we didn’t get a full post backup – our records stretch back to the 26th.
The second item on our agenda was to check to ensure that we had some way of rebuilding our posts. Thankfully, most of our daily news updates were in Google Docs, and the remaining items were accessible via our Tumblr Page, so we have everything ready to rebuild.
Unfortunately, since we were on a VPS system, the server was 100% our responsibility – we don’t have logs of who did this, exactly, and we were face with an option of spinning up a new server and hoping it doesn’t happen again… or changing hosts entirely.
What We Did
Unfortunately, this exposed a very real reality to my world – I’m one person – I work during the day as a software engineer, and while I can administer a server, I really just can’t keep up the pace for the Herald. I don’t have that ability to be there, 24/7/365 when things go wrong, nor do I have the resources to spend to monitor situations moment-by-moment throughout the day.
This has prompted us to move to a new host, who will manage our installation, and the site as a whole. We’ve signed on with a managed provider, who will keep our site’s core infrastructure safe.
Going forward, we’ll continue to do our best, and strive to be that place where we’ll talk nerdy to you day in and day out.
What’s Left to Do
Over the next few days, we’ll be restoring what we can – bringing articles back that were lost, and what have you. We’ll be recording Nerdy Talk tomorrow, and we’ll have our regular Herald Anime Club piece tomorrow evening.
Unfortunately, though, we’ll be taking a rain check on AniWeekly and Media Create, due to the amount of time they require. Our energies will be focused on getting our house back in order, so we can continue to bring our very best your way.
I don’t say it enough, but… thanks, for letting us do this for seven years and counting. :)