TEC didn't automatically return from its Monday morning reboot, today, and instead had to be coaxed back up by hand. All should be normal again, with no data lossage.

The restore of TEC we did this morning has held steady for almost 6 hours of use, which is far better than any of the troubled restores we did. So, we're hopeful now that we're going to be able to stay with this one. Though the downtime was extended as we picked through backups, total data loss turned out to be only 2 or 3 hours.

We are testing out a restore of TEC that I'm highly suspicious of. We should know in the next few hours if it's stable or not, but be aware that we'll probably need to reset this one.

Unfortunately, after multiple tests with multiple TEC backups, we've been unable to restore it to a working state. It's likely that something else is going on than just data corruption, but it'll need to wait until tomorrow at this point, so that I can see if any other engineers have ideas and/or consider more extreme measures, like pushing the game to a different machine.

After some more tomfoolery, the game is back and looks stable. We managed to reduce the rollback from 34 hours to 10.

After 3+ hours of fighting, TEC is back up. Unfortunately, all of our backs following Monday morning were corrupt, so we had to rollback about 34 hours. We'll be looking into reasons for this in morning, but for now enjoy the game.

Sadly, it looks like the core dump that occurred at 2:55 corrupted the database. We're going to be rebuilding from the save around noon.

We are aware of the current problem with TEC, and are currently trying to cleanly restart the server.

Back up, from the 1pm PT backup.

TEC got stuck for some reason this evening and unfortunately would not restart cleanly. We're now restoring from backup and will try and restart the 1pm (PT) database.