The prognosis on the database upgrade from last night is... not good. In addition to the bug that was uncovered this morning where posts canít be edited, we now appear to be experiencing table deadlock, and it is grinding the whole site to a halt. The culprit, as usual, seems to be search.
I have just upped the IO capacity of the database somewhat, as we were bumping up against some hard IO limits and then getting throttled. Iím not entirely sure this is the answer, though. I am monitoring the database to see if it starts grinding to a halt again.
If the upgraded IO capacity doesnít solve the problem, I will most likely have to revert to the database as it stood last night at 6PM PDT. That would be... a disaster, and not something I say lightly. We would lose almost 24 hours of data, including any new user registrations that happened during that time. But it might be our only option.
Fingers crossed. The next few hours are going to be tense.