Modern Vespa

We're having some lingering issues with the server, specifically the database. It's partly because I'm running a big (actually huge) re-index process in the background, and it is taxing the database a bit more than I expected.

I've been rehearsing this particular re-index for days (a week?) and it went smoother on a test database, but then nobody else was using it at the time. Whenever I stop watching the log output, the process starts slowing down until it brings the whole site to its knees.

So, sorry about that. I've scaled back the re-index a bit, biting off smaller chunks at a time so the server doesn't choke.

There are a couple of other issues, too. There are a couple of places where the new database is stricter than the old database, so I am finding occasional query failures in the logs where I just need to tighten up the parameters a bit. Put another way, some of the database queries were sloppy

The final issue is that someone (or several someones) are running a lot of searches at the moment. No fault there, I'm not asking anyone to not search, but searches being the most strenuous thing the database does, combined with the other issues, means I can see more than a few failures in the logs -- a LOT of failures, actually. Hopefully that will improve as I scale back the re-indexing.

One positive note, too: I've upgraded the underlying storage system used by both the database and the main server itself. Amazon AWS has recently rolled out upgraded SSD drives (gp3) with much better throughput, and I am taking advantage of those everywhere I can.

Thanks, yet again, for your patience.

"Whenever I stop watching the log output…"

So true.

That may explain a few things. I was on the pissed off thread and I threatened to go and punch as many Christmas lights as I could to work out my frustration. Then I suggested that the moderators give me a title for the most light bulbs broken in the shortest amount of time. Just innocent stuff. And then I get this picture and message

I took the site down temporarily when it was clear things were really going sideways. I had done my re-index rehearsals on test databases running on the new gp3 storage modules, but the "real" database was still running on gp2 storage modules. I inadvertently maxed out the capacity of the gp2 storage modules, and AWS started throttling my IO data rate -- making a bad problem even worse. I had to shut everything down and quickly upgrade the database to gp3 storage modules to sidestep being throttled.

Hence the picture of Patrick.

I thought it was some kind of a prankster

Definitely not. Unless I'm the prankster, playing jokes on myself.

I have noticed a roundel on the thread page. Does that mean that I am anything special or any inpending award?

That's the link that takes you to the smashy light things.

Thanks. Well that says a lot about me. I used to be a huge Christmas lights smasher. 12 step program and everything. And now that it's back I have relapsed abysmaly.

Or, FWIW, refreshes the smashy light things.

That's new isn't it? I don't remember seeing it yesterday.

Heh - if you draw your finger (or mouse pointer) across them they disappear. So no rapid bulb-smashing.

It is new, along with a new search icon.

No, they really do smash.

Doesn't the roundel duplicate the functionality of the big "wasp" directly above it?

I like the streamlined magnifying glass logo!

Yes, but so did the word "Forum" that it replaced. Trying to save some room on the menu bar, since there are temporarily two different search options.

I'm finding submitting posts is taking an age, every time. This inevitably leads to the 'insanity' message due to my impatience or quick editing of the (still open) compose window...

^^^^ 16 seconds.

You're not imagining it. It's been happening since we switched over to a new database, though not consistently. I mean, it's always been slow, it's just slower now than usual.

I suspect there is some SQL somewhere that needs to be optimized for the new database, as there are subtle differences in optimal key usage from the old DB to the new. I've already fixed a few of these long-query problems, but have not tackled this one yet.

I kind of suspect that I'm going to discover that the issue lies in updating the search index of the old search system, which has historically been the biggest hit on posting speed. I really want to jettison all that code, now that a new search is in place. But likely it's too soon.

(Sidebar: the new search system does all of its indexing in a background process, separate from user interaction. I may be thick, but I've learned a few lessons over the years from this @#%^ forum software).

That makes loads of sense. I wasn't going to accuse the new database indexing, as I was sure you'd have avoided that chicane.

To be fair, it wouldn't be the first time I've been surprised by the side effects of my own changes.

I turned on a diagnostic mode that shows me a duration and full explanation of every single SQL query necessary to build each page, all in an effort to track down this long-posting-wait bug.

Guess what's not happening any more?

Snerk...

edit - But that was ~8 secs.

The plot thickens!

Edit: Almost instantaneous.

I've had a slow-post a couple of times, but the database query stats were very fast. In fact, the whole page output was under 300ms or so.

HOWEVER

I think the stats I'm seeing are on the redirect to the topic to show what I just posted, and don't reflect any of the activity prior to the redirect. So I've resorted to putting a slow-query check inline for the whole forum, dumping out lots of cruft to the logs in the hopes that I can spot the problem.

This might be the culprit. It would disproportionately affect Jim and I -- and also stickyfrog and OAD, as the query time is directly dependent on how many posts must be scanned.

I was wondering if it had to do with the total number of posts...

^^^^ ~8 secs again.

It's not consistent, though.

Hunting through some logs now.

I am retired so I do things slower now.

Hah! Well, I don't want to waste any more of your time than I have to, so I'm going to keep hunting down this bug.

I'm not 100% positive that the flood control query is wholly responsible for the slowdown, but the way it goes about doing it (vintage phpBB, btw) is 100% shite and should be fixed. Scanning for the maximum post time among all of a user's posts is just... that's just poor planning.

Instead, I'm using redis (our in-memory cacheing system) to keep track of the last post time for each session. It's a small amount of data, and when the session expires, the record disappears. Which is perfect.

(The redis cache has really revolutionized how the MV underpinnings work -- it is soooo nice to have a robust in-memory key store with automatic expiration of keys).

Anyway, posting already feels faster for me. For people with only a few hundred posts, there won't be any noticeable change.

Have you seen / felt the long-post-delay in the last 24 hours or so?

Let me see...

... as of now it's very, very quick.

I am thinking the flood control code was the culprit.

I've also been quietly optimizing a few other queries today, based on the flagged queries I'm seeing in the logs. I've had a couple of medium-sized wins today -- queries that were taking half a second or more reduced down to a few milliseconds.

A couple of issues this morning. After I put in my username and password I get the first page and it looks all right but as soon as I hit the general discussion the screen gets misaligned. I'm including a couple of snapshots. Anyway I thought you might want to know

Thanks, Max. See this post and the few after it for resolution.