Modern Vespa

Each day, and each week, and (sigh), each year, I know slightly less than I did the day / week /......
.....um....what was the question?

Is this Tuesday?

It's Thursday, actually.

Im thirsty too, let's get a beer.

I, for one, and I am sure most or all are thankful for and appreciate you jess for all you and the moderators do to keep the lights on in this place, even more so with the ever increasing headaches.

Now, if we can get AI to battle AI/bots, that will be something.

There are actually some services that filter out bots through AI (or really ML, "Machine Learning"). These are generally well beyond our meager budget though, and I am very reluctant to make MV dependent on any more services than I absolutely have to. In fact, it's probably cheaper to just absorb the extra bot traffic (with the added bandwidth and compute costs that come with it) than to pay for the service.

As a fellow traveler who has spent waaay too much time in the last week trying to convince some of these more expensive tools that we are indeed getting hammered and that it's obviously bots, I think your approach has been even-handed and inspiring

Thank you, sir. This means a lot.

Oh, I almost forgot the obligatory XKCD cartoon:

Hang onto your hats, everyone. This is the worst attack yet.

I might have to block entire regions of the world for this one.

China and Hong Kong are now completely blocked at the CDN level. So the server will never even see the requests.

Yes, this will likely affect genuine users. But I've got to protect the server from the onslaught of abuse coming from that region of the world.

Singapore and Philippines, which are both used as data hubs by large Chinese conglomerates, are still problematic. I'm only seeing a small amount of malicious traffic from those two countries at the moment, but we have in the past seen floods of traffic from both of them. I really don't want to ban Singapore, as it is definitely part of our audience.

And magically, we have calm on the server again.

You're a wizard!

Thank you for your diligence.

I might have spoken too soon.

Okay, so...

There are a lot of things happening simultaneously. We have a lot of different kinds of bots that have been hitting the server pretty hard over the last hour, but I am seeing several different bot "styles" if you will -- what they're asking for, how they go about asking, and from various origins.

There was definitely a strong presence coming from China. Most of these large Chinese concerns were already blocked, so none of them were getting any data. But the server was still needing to respond (and reject) all of those requests, which is still a load. When the requests are coming fast and furious, I get concerned.

Those are now all blocked at the edge, using CloudFront region blocking. This allows me to block the request much closer to its origin, using the CloudFront CDN we were already using. This means the server won't see any of those requests anymore. Along with the legitimate ones from China and Hong Kong (however few there were).

But I was also seeing a ton of different requests, mostly from the US. And I can't really block that region.

As it happens, CloudFront supports "edge functions", little bits of JavaScript code that can help filter incoming requests, but at the edges of the network, near the origin of the request. This means that the javascript edge functions get pushed out to all the various geographic regions around the world, and are invoked for every single request intended for Modern Vespa in each of those regions. Distributed filtering, basically.

So I added a couple of simple javascript checks to filter out the most egregious requests I was seeing -- two specific JA4 signatures, and some jackass that thinks it's a good idea to make thousands and thousands of HEAD requests to the server (not GET, not POST, but HEAD) -- which the server never supported anyway. But now the server will never see those requests, because they are halted at the edge.

And, for now, I have another brief moment of calm, for however long it will last.

Fully understand budget constraints and not wanting to become dependent on any more services than necessary. I was obviously thinking about reducing human work and the comment was somewhat tongue-in-cheek.

Saw your latest post in this thread. At the risk of stating the obvious, this crap is never ending…

Thanks again for all.

Today, I pushed an entirely new screening system to the server.

When we get a request from a client that we don't know, or that we strongly suspect is a bot, we give them a screening page (the "I am not a robot" button) instead of the requested content.

As simple as that is, it actually thwarts a ton of bots that are largely too stupid to figure out how to navigate that dialog. There are some additional measures in there as well, involving javascript and the DOM and cookies. But for the most part, what we've had up until this point has been pretty effective.

Unfortunately, that "I am not a robot" page gets shown to a lot of clients that are (probably) actually human -- somewhere around 10 percent of initial requests to anonymous clients (i.e. clients who are not logged in) end up over-screened, and subsequently behave like actual humans. This is something I've been wanting to address for a while now.

The change I've just pushed out attempts to mitigate that, by doing some initial checks on the client during that screening phase -- before it even asks if they are human -- and in some cases will decide automatically to forego further screening. There is still a slight delay on that initial request (as there is some back-and-forth between the client and the server) but that can be as little as 1 second before the auto-screening completes and the requested page loads. In some cases, the human (if the client is in fact human) might not even notice it is happening.

Will this improve the over-screening percentage? Not sure yet. I'm still gathering data.

And watching the logs.

I am forbidden again using Firefox on my desktop, but with same external IP address. Vivaldi still works though, and I post this from Vivaldi.

What is the specific message you are seeing? Is it 403?

Okay, so this would appear to be the sequence that triggered the ban (ignoring the access from google in the middle of it). For whatever reason, your client made repeated requests to load the login page, but (again, for whatever reason) did not execute the javascript embedded in the page.

I really don't know what's going on with your web client -- and I think I asked before -- but do you have javascript turned off?

I have removed the ban on that client, but I can't guarantee it won't get re-banned if the client requests multiple pages in quick succession like that.

Sorry, Jess. It was Forbidden. It worked fine w/ff on the tablet, so was surprised when it would not on the desktop. Thanks, I'll check soon.

Let me know, and email support@modernvespa.com if you get stuck.

I've done some more digging -- quite a bit, actually. And spelunking through the logs. The good news: I was able to replicate your bug -- which was totally not your fault, nor the fault of your browser. It was an edge case, for sure, and it was exacerbated by a tiny detail (also not your fault) but that you might want to remedy: I think you are using a bookmark to log in to MV that is:

http://modernvespa.com/forum/login.php

Which, to be fair, used to be the login address -- but it has changed to:

https://modernvespa.com/account/login

The old one will forward to the new one, but in your specific case, under exactly the right circumstances, it triggered a whole series of failures that (again) were not in any way your fault.

(I would still advise changing your bookmark at your earliest convenience).

In any case, it should be all fixed now, and I am reasonably confident that this particular bug won't happen again (until I break something else).

Thanks for your patience.

I'm having the same problem with Firefox today. I checked the login address as above but I get the 403 Forbidden message. Same with Safari. Only on the desktop, though. Logged right in on my phone and tablet.

Let me check the logs.

Okay, you got me. Same exact failure as Syd was experiencing, in spite of what I thought was a solid fix.

I am scratching my head on this one.

I will dig deeper.

It looks like there was another edge case that I was not accounting for.

The login page has some extra security around it, due to bots attempting credential-stuffing and brute-force attacks. So we require that before logging in, the client must have already passed various tests to determine that they are actually human (and not a bot). This greatly reduces the number of login attempts that actually get to the point of testing credentials (which is a very good thing).

So if a client requested the login page on their very first request of a new session (such as would be the case for someone using a bookmark to access MV's login page), then at that moment, we are not yet sure whether the client is a user or a bot -- this is known problem.

My original solution was to redirect the user back to the start of the login page a second time, so that we would have another chance to determine if the client was human or not (which takes a page or two to figure out -- and is a long and complicated story all by itself).

In my previous attempt at a fix, I made sure that the login page was "screen-able" -- that is, that the system properly understood that it could enforce a screening test before delivering the login page. This allowed screening to actually take place in between attempts to load the login page, at which point we would find that yes, the client actually reads like a human, and then the login process could proceed as normal.

What I didn't account for was the case where there is no reason to screen the client. That is, if their IP address was completely clean. In my development environment, screening happens on new sessions 100% of the time, because I am often testing the screening process itself.

But on the production server, there are often clients that don't require screening, and so my assumption that we would screen the client and determine they are human did not actually happen. And as a result of that, we went into a redirect loop, making repeated requests for the login page until the system flagged the client as a bot and banned them.

My additional solution here is still to re-load the login page if we aren't sure the client is human, but force screening to happen no matter what in between requests for the login page. This effectively means that people who typically reach Modern Vespa via a bookmark specifically of the login page are going to get screened. Every single time. And that is a necessary evil, because the login page is such a tempting target for bots that are genuinely malicious.

My official recommendation: don't bookmark the login page. Bookmark the home page instead. An unknown client requesting the login page on their very first request is indistinguishable from a bot, and in that case we must screen before allowing the client to test credentials.

Better yet, don't log out. It isn't necessary. MV will retain your logged-in status for up to a whole year, if you don't clear your cookies.

That said, I again extend my apologies for the trouble, and thank you for your patience.

I don't even close the page - I have it pinned in Chrome next to GMail, Calendar, and all that

Completely reasonable! That said, closing the page (or not) doesn't make a huge difference -- after one hour of inactivity, user sessions are automatically ended. However, if you don't log out (and don't clear the cookie) a new session will automatically be created the next time you request any page, without needing to log in again.

The mechanism behind this is a login token (in the form of a cookie) stored in your browser's cookie cache. Every time you request a page, your browser sends the login token to the server, which then uses it to either (a) find your current session (if there is one), or (b) start a new session for your account.

I use a password keeper, and it was set to the login page. I edited the address to take me to the home page. I tend to log out and shutdown whenever I'm not at the desktop, a habit from the Cuckoo's Egg days of dial up and remote logins. I have an Apple silicon Mac, and don't really need to do this, but it's my habit pattern. I also delete cookies, history, and selectively block trackers, and have my Preferences to Private browsing. Plus, I use a paid version of Proton VPN. I'm not at the tinfoil hat TOR/anonymizer level yet, but you can't be too careful these days.
That being said, this is a great forum and you do a fabulous job. Thanks for knowing the bits and bytes so we don't have to.

All completely reasonable, and entirely personal preference. Just wanted to mention that it's not strictly necessary -- the stakes are fairly low

But if anyone feels strongly about logging out, that's totally okay too. My pleasure. Thanks.

You are correct. I use the Quick Dial extension to FF and added that exact bookmark. I'll fix that asap.

Got a 403 Forbidden / Banned IP message. I use a VPN service, I got a message to verify I'm not a bot but I was presented the mentioned error. Now using a different server / IP. Previous one was 194.116.208.155...

I see the messages in the log, starting at 19:54 UTC. The requests were shunted over for screening (because you weren't logged in at that point, and it was a new session) and the server didn't get a response for any of the screening requests.

Did you actually see the button that said "I am not a robot"? What happened when you clicked it?

It happened again but only through Chrome on PC, it says "Reloading stale session", I get the "I am not a robot" button but pressing it reloads the page and then I get the Banned IP message. Erasing cookies, etc. didn't help…

Oh, interesting!

Is it possible your computer isn't synced to an accurate source of time on the internet? The message you are getting happens when the button isn't pressed within one minute of being presented -- or at least, when we think it was presented. This might be because your clock differs from the server clock by a significant amount -- which is something I hadn't seriously considered until now.

Time and date synced up fine...

I've removed the time check from the screening code. Give it another try.

(And if you're still blocked, let me know the IP address you are blocked from and I will unblock it).

Still getting the Forbidden message through Chrome. IP is 194.116.208.54

Heh. You managed to get the entire ASN (AS203020) banned for all anonymous traffic.

Should be unblocked now.