Modern Vespa

Perhaps a temporary redirect to a very very very large Rick Astley file? A proxypass back to the originating IP address? Just keep their connection open and see how long you can make them wait?

All excellent suggestions.

Though right this moment, I am distracted by repeated coordinated attacks on the site, that have several times overwhelmed our modest resources. I need to find a solution for this swarm that seems to be coming from all over the world simultaneously.

I've just had to reboot the database, as it was struggling.

Here's one for the bots:

Word may have gotten out that you run a darn good forum. I just saw that go by.

Maybe the internet is just growing, getting faster, and is more populated by hungry bots and scrapers; maybe we could get you enough cups of coffee for a bigger hammer to deal with it all?

Traffic segregation could be another way, like shunting all non-logged-in pages to static/cached versions or something, refreshed at low frequency. Are there certain places the bots are getting that are particularly stressing the server, some kind of expensive page, or is it just too much overall?

Yeah, that's a possibility. The demand for data seems to be insatiable these days. We've been able to get by on modest hardware for years, but I'm wondering if that is sustainable. Most of the requests are for topics, though there are some bots that seem intent on reading the wiki incessantly and some more that really like to aggregate lists of member profile pages. Oh, and some that seem to hit the topic search pages on their first request, but I've just today put in a defense against that.

We cache the rendered posts (as HTML) in the database already, and reassemble the topic from the already-rendered posts. This saves processing power (BBCode is expensive) but is still database-intensive, and that's where the bottleneck is right now. Unsurprisingly, the database is also already our largest single expense, and so increasing the capacity is going to be a bit pricier.

The server itself -- the raw processing -- seems to be up to snuff. When it doesn't have to hit the database, it can process a surprising volume of requests without breaking a sweat. Just a few minutes ago I watched it being pummeled by requests from the Alibaba data center (all of which are blocked by IP address) and it handled several thousand requests in a minute or so (see picture below straight from the logs).

The ones that have been problematic lately have been very swarm-oriented -- everything will be humming along fine, and then all of a sudden we get hit with requests from all over the world at once. I've been considering rate-limiting the number of new anonymous sessions across the board, and going into a protective mode at some yet-to-be-defined threshold -- kind of a temporary shield that would block additional anonymous sessions for a minute or two, enough for the database to catch its breath.

I'll investigate tomorrow. That shouldn't be too hard to implement.

Ugh--that's disheartening. And just plain rude. It seems like people have figured out a way of making DDoSs profitable without the whole extortion part slowing it down. Saint Jess of MV--we're rooting for you! If we see the picture of the Vespa being eaten from the inside out, we'll know you're taking a well-deserved break from this silliness.

I genuinely don't believe it's intended to be a DDoS campaign, even though it is effectively a DDoS campaign. I think the people running these operations -- which I still believe are attempts at scraping the site -- are just complete amateur clowns when it comes to putting together a scraper.

Or, put another way: never ascribe to malice that which can be adequately explained by incompetence.

Ars Technica ran an article recently that (superficially) describes the situation I've been facing for the last couple of months.

https://arstechnica.com/security/2024/04/everyday-devices-are-used-to-hide-ongoing-account-compromise-campaign/

In the cases described in the article, the target is credential stuffing. I'm not seeing very much of that (no more than usual, anyway). But the swarms that I'm having the most trouble with appear to have the same general MO as what's described in the article -- requests from everywhere, all at once, using what are almost certainly residential and mobile IP connections that are nearly impossible to block.

...and, we are currently under attack. New swarm. Mostly US IP addresses this time.

And another meltdown. Not entirely sure what caused it -- the only thing suspicious was a New Zealand IP address requesting the same topic over and over again. I wouldn't think that would melt down, but... not seeing another obvious reason.

(And no, I don't think it was znomit).

One of the unexpected benefits of putting the MV server behind AWS CloudFront is that CloudFront will give me a bunch of additional data with every request, in real time, for free.

We already had access to decent GeoIP data (through the MaxMind GeoIP database) but CloudFront can automatically attach that data to each request, meaning the MV server has to do less work (and is presumably faster for it).

We also get the ASN for the IP address, and the JA3 fingerprint. Neither of these things are useful for uniquely identifying a client (or a bad actor) but both of them give me some additional options in tailoring an automated response to an attack while affecting as few clients as possible. Combined with a number of other signals (i.e. registered or anonymous) I can automate restrictions that I would be otherwise uncomfortable with.

So that's cheering me up today. Oh wait, wrong thread!

When there's an embedded Youtube video on MV, I get no sound playing it regardless of the embedded volume control, unless I use the "watch on Youtube" option.

I *think* this is specific to MV (BICBW), as other sites with embedded Youtube videos play unmuted just fine.

All audio options in Windows and Chrome are turned ON.


Using Chrome on Windows 10.

Okay, I'll investigate. I think I'm getting audio on Safari, but I'll double check.

I've been trying to reproduce this on Chrome (on a Mac) and so far have been unable to. I get audio regularly.

The only way I can get the behavior you describe is if I specifically mute modernvespa.com in Chrome.

I'd really appreciate it if some more people (especially on Windows) could weigh in here.

Well well well - I eventually found the setting buried six layers down, and MV was the only site that had been refused permission to play sound. I have no idea when or how that happened...

So please accept my apologies for wasting your time chasing a phantom!

No worries!

Maybe you turned it off one Christmas?

I wondered that - but it's far more complicated to adjust that setting than to just mute all sound while rummaging around in MV. It's not in my nature to take unnecessary extra steps!

In the never-ending battle against the bots, I thought I'd show you what an attack looks like. Fortunately, this one was mercifully brief, lasting only a few seconds. As you can see (maybe) the forum ramped up the defenses after a few requests and started denying requests, which seems to have convinced the attacker to seek greener pastures elsewhere.

Prior to me putting in these defenses, one of these attacks would go on for several minutes or even hours, in some cases. And it would often bring the database to its knees, as we are just not equipped to handle that kind of traffic surge.

All of these requests (except the couple of search engines that were active at the same time) originated from China, using a large variety of IP addresses. However, all of the IP addresses in question originated from only two network entities:

ASN 4134: CHINANET-BACKBONE No.31,Jin-rong Street, CN
ASN 140061: CHINANET-QINGHAI-AS-AP Qinghai Telecom, CN

This made it fairly easy to just shut down any requests originating from one of those two ASN networks for a short time.

Now that's a pretty looking log! Color-coded and everything

Do you consider yourself over the hump in this battle, now that you can identify and reject so quickly? What is the fingerprinting method that you're using (if not secret sauce)?

Thanks! I spend enough time looking at the logs that it was worth adding color to make it (slightly) visually easier to distinguish different kinds of events. Mostly. The reality is that it's a constantly evolving struggle. I've put many defensive mechanisms in place over the years -- some of them have continued to be useful, and some of them have become obsolete as the adversaries have evolved. There are a variety of different identifying markers visible in the log. The most obvious is IP address, of course, followed by the ASN. The next column -- 16 characters of base64 -- is the client fingerprint, which is a hash of the IP address and the user agent. The next 16 characters of base64 are the sessionID, which is only persistent for a few minutes for anonymous clients.

Not shown in every row but alluded to in some of them is the JA3 fingerprint, which identifies the TLS/SSL fingerprint of the client. This isn't granular enough for identification but it can be used as a backstop form of rate limiting -- if I get a whole bunch of requests from anonymous clients using the same JA3 fingerprint, I start rejecting them once they reach a certain threshold. Generally, this rejection state will only last a few seconds, as the measurement period is very short. Unless they keep attacking, in which case all anonymous clients that share the same JA3 fingerprint will be rejected.

None of these measures apply to genuine crawlers (Bing, Google) or to registered users. While there are rate limits for both, they are much more generous.

This was the biggest attack I've seen in a while. The requests were coming from a very diverse set of IP locations, all over the world, simultaneously.

Nasty.

Fortunately, the client fingerprint was identifiable for all of these requests, and (almost) all of them were rejected.

What on earth are they trying to achieve?

No idea.

It was a slightly rhetorical question - I'm imagining an evil army of Ring doorbells, fridges, cat-flaps and lightbulbs ganging up on the forum that stops their owners from paying them the attention they feel is their due...

Heh. Maybe! There were so many different clients that I didn't bother chasing any of them down to try to figure out what kind of device they were originating from. And likely I wouldn't be able to tell.

But it was an impressively broad attack. There are rarely more than a handful of IP addresses in one of these flurries, but this one easily had many hundreds of IP addresses from all over the world. That can't be easy to coordinate without a botnet.

Most of the requests were for member profile pages, too. Usually it's primarily topics, with a bit of wiki and member profiles mixed in. This one was clearly after member names.

What DID Jess fix next?

I just noticed that when I hover my mouse over a Tab opened to MV I get a little menu of Icons.
When did they appear?? And what do they do??

Edit:
Nevermind. I discovered it was not an MV thing, it does it everywhere I go.
Must have been an Opera update.
(But I still don't know what they do.)

Sounds like you might be talking about this new feature?

People seem to really dislike it. I haven't used Opera in... 20 years? So I have no insight here.

Been getting this a bit today on the laptop.

Using private relay on the MacBook.

Hmmm. Not entirely sure why that might be, but there are a couple of possibilities. How long does it last in that state?

Last one was about 5-15 minutes.

I can probably diagnose this if I am at my desk when it happens, but I'll need to know the specific public IP address that you are using at the time. And seeing as how you can't post when it happens, you'll have to email it to me. And I'll have to notice. Which... seems improbable.

But go ahead and email support at mv if it does happen, and I'll give it a go.

A quick question about the domains: modernvespa.com, modernvespa.net, modernvespa.org.

I was using google to try and hunt for information on MV and, after adding a particular filter, I noticed that I got modernvespa.org results.

Poking around a little, it looks like modernvespa.net 301s to modernvespa.com, but modernvespa.org does not, as well returning pages and links with .org in the base urls.

I was wondering if this was a "feature" (I can have separate accounts pretty easily!) or a "bug"?

It's… inadvertent. The domain modernvespa.net is used exclusively for static content and user-uploaded content (e.g. images). This provides a slight benefit in that clients won't send modernvespa.com cookies when making requests, and modernvespa.net has no cookies of its own, which theoretically improves latency while fetching images and things.

The domain modernvespa.org gets used (by me) for staging various things and experimenting with configurations. It really shouldn't be live, or visible, but at the moment it is. Which is really just a configuration oversight.

Please don't use it, it might disappear at any moment. It might also change to something unstable.

(But thanks for the heads up — a reminder of something I should fix).

I think I found a bug. Trying to address a specific user with the @ symbol doesn't work with his nick, probably because he has a space between the first and second letter…

Not a bug. Legacy usernames that contain spaces use an underscore for tagging.

If you visit v oodoo's profile you will see the proper @ tag near the top.

https://modernvespa.com/members/v_oodoo

The same thing goes for usernames in URLs ^^^

I've noticed something that only happens on MV, and not other websites I visit.

I have started using Grammarly to help with my writing for school. I've installed the Grammarly plugin in Chrome on my MacBook Pro. When I compose a post here, the underline things that Grammarly generates end up not being connected to the writing at all, and instead end up all over the page. I've added a screenshot to show you what I'm talking about.

Like I said, it doesn't happen on other websites. It's not that big of a deal to me, but thought you should be aware.

Interesting. I'm not sure what that's about, actually. We're not doing anything especially weird with the text form, aside from resizing it for the window. I think it's possible they are missing the resize event, but I am honestly not sure how to remedy that.

I'll look at the code and see if anything jumps out, but I might not be qualified to diagnose this particular issue.

monogodo I would be curious if it happens with the Grammarly Safari extension. Not suggesting you switch to Safari, just as a test.

This is me composing a response to you in Safari, with the Grammarly Safari extension active. There are no indications that Grammarly is doing anything. There are no underlines under questionable words.

I've copied my previous post text and pasted it below to see what comes up here.

I've noticed something that only happens on MV, and not other websites I visit.

I have started using Grammarly to help with my writing for school. I've installed the Grammarly plugin in Chrome on my MacBook Pro. When I compose a post here, the underline things that Grammarly generates end up not being connected to the writing at all, and instead end up all over the page. I've added a screenshot to show you what I'm talking about.

Like I said, it doesn't happen on other websites. It's not that big of a deal to me, but thought you should be aware.

--------

Ok, I didn't have it turned on. As you can see, it is doing it in Safari, too.