It actually looks like I caused a screw up because I misidentified a DoS attack, but things cascaded from there.
I recently did an update that was problematic and was causing the 502 and 504 errors and putting big strains on the server. This was a week or two ago, but I got it fixed. Then I did what I thought was a minor upgrade yesterday, and then we had the problems of the evening, which without looking under the hood looked eerily similar to the problems I had a week or two ago.
Because I have been pretty slammed with work I assumed it was that minor upgrade resurrecting the past problem, so without investigating further I rolled back to a backup - but that backup was corrupted. It wasn’t until I got to a 3 days old backup that I discovered a backup that was valid (I do them daily).
But while I was going through that process I thought I had nuked the entire site, which was not a happy feeling - I have some older server images so I guess all was not lost, but at 3am it sure felt that way.
So, once I got the three day backup restored, the 504 errors began, server load was REALLY high and so I investigated further. Tons of hits on the site from Russia. I guess all of our DJT jokes caused some blowback.
So… I could probably retrieve the past couple of days posts with a little work, but honestly don’t think it is worth it given the pay. I’ve also done a few things I should have done from the get-go but didn’t because this is mostly a hobby to me, although I feel an obligation to keep things running here smoothly to all you donors.
I’m running our domain through Cloudflare now to help with attacks - I should have done that from the get-go. I’m modifying the backup frequency and where they are stored and will run tests on the backups periodically. I’ll also schedule some server images as a backup to the backup through Amazon Web Services. I’ll also not rush to conclusions in the future and look at the logs before I take action.
In short, a comedy of errors really that in the professional world might have resulted in my termination (which is why I typically leave the development to my developers) but I’ll do better next time. My apologies.