Board Downtime

[Lord Glenn]

As many of you are aware, the boards were down yesterday afternoon and remained down until a short while ago. The boards are obviously now back up, though we were forced to restore the boards back to the backup from a day or two ago, so some posts have been lost.

Further information will be provided later in the day, once we've gotten everything sorted out.

We thank all of you for your patience during the downtime.

[Superbus]

Here's essentially what happened:

We ended up having an error last night about 1830 EST:

1)
8: undefined index: emoblack
File: /home/.bandit/fessftpguy/forums.grandbell.net/Themes/dilbermc(followed by two characters that I can't remember)/fess(another four characters I don't remeber)/Index.template.php (main sub template - eval?)
Line: 11

There was a similar one later for reading threads. I'm not sure what happened, but I think someone was fucking around with the templates; I'll find out who, and they'll be dealt with.

I tried to roll back the forums a day on the FTP side, but that didn't seem to do anything. A second day didn't help either, so the issue was with SQL. That's where the fun started.

I rolled back a day, telling the SQL restoration programme to make my old tables the defaults, while moving the new ones with a timestamp, so they're there in case I need to restore them. The programme - on Dreamhost's end - ended up making the current (broken) tables more broken by putting the timestamp on them. I went back another day, having them done another way, and it affixed THAT time stamp to the same SQL tables; by now, my tables have like three different timestamps on them, and look ridiculous. I finally decided to tell it "OK, restore from two days ago, and drop the old tables". It didn't do this. Then, an hour later, I learn that it dropped my old tables PERIOD. No restoration, I had an empty SQL host.

At this point, I'm pitching a right fit, and put a second ticket into Dreamhost, basically screaming for bloody murder (I had one requesting a call back in earlier as well). They didn't get to me until about 1400 EST, telling me that they were loading everything in, but it was taking awhile (the total usage of my SQL is about 5GB, so that's understandable), and they wanted to let me know that they were on the case once the manager woke up and said "hey! We have a customer about to have a coronary!".

So that means that we had a twenty HOUR gap between my initial SQL ticket and first contact; I put my second, more terse one in seven hours after the first. That, I believe, has been dealt with on both of our ends, especially after a rather scathing survey on my part.

The end result is twenty hours of downtime and two days worth of missing posts. I sincerely apologize about that. I'm not about to change hosts (down, Blacken!), because overall, for years, Dreamhost has been good to me for the most part; you don't break good business relationships over one fuck-up. As for what we can do to prevent this in the future... I'm not sure! The ultimate failure was on Dreamhost's end, but I'd like to find out exactly what the fuck happened with the templates and those files that were returning broken PHP code. This is going to be a shitty next few days for my staff until I get an answer.

Either way, we're back up, and again, my apologies.

[Blacken]

OK, so who was fucking with Themes/dilbermc/index.php? Because whoever is fucking with it (and it's an admin, because nobody else has the ability to edit it) apparently didn't fucking realize that--hey!--it's been hacked all to hell and changing it probably isn't stable.

Argh.

EDIT: So somebody had FTP who shouldn't. winrar.exe.

[Superbus]

OK. The good news is that now, I KNOW it wasn't someone fucking around with things. The bad news is that I know this through the fact that we fucked up AGAIN. Same errors.

This time, with help from DMAJohnson, I went through, and basically commented out any lines that gave me bullshit (Ed, I'd like you to look over those later, if you have time up in WannabeAmericaLand). We're back, though I'm noticing a few images missing.

I need everyone to let me know if anything else is wrong. But right now, we LOOK OK.

EDIT: Ed, this is for you: I made backups for both files that needed editing: MessageIndex.template.php (dilbermc) and Display.template.php (default). Those are the files that are fucked up. The current ones are what we're running now.

Here's what I wrote out:

Message:

11, 185, 188

Display:

321-325, 328, 329, 385

[Blacken]

Email them to me along with any errors; I'll look tonight after the Sox game.