Modern Vespa

You and at least one other person, who I've been conversing with via email.

Apparently, an unrelated change I made a few days ago had some unintended consequences, but only for a handful of people. I'm theorizing that something about the way I was setting cookies was unpalatable to a small handful of browsers, but the exact details aren't clear.

I bet those browsers like nuts in their oatmeal cookies.

Wait a minute. Three Bob's had the same problem?!!

Yes. And this pattern is not lost on me.

Make a note when it gets to nine bobs.

I'm thinking that this is probably what was tripping me up:

https://developers.google.com/search/blog/2020/01/get-ready-for-new-samesitenone-secure

I hadn't been setting SameSite on any cookies up until two days ago. I needed to start setting SameSite=Lax for a different feature (still in progress) but didn't want to disturb the existing cookies, so I made SameSite an option that I could configure depending on context.

Except, when I wanted the old mode, I ended up setting SameSite=None, instead of just not setting SameSite at all. Semantically, I would have thought them equivalent, and indeed in my testing on Safari, there was no difference in the resulting cookie cached in the client.

But this appears to be a problem for Chrome.

Sorry to bother, jess, but I had a question about the "My Notes" functionality.

After logging in, I go to "My Notes" (https://modernvespa.com/forum/notes.php). From there, I add text to the textbox and click "Save". I get a response page that reads: The middle line includes a link back to https://modernvespa.com/forum/notes.php where, following, I have a blank page and an empty textbox again.

I may be missing something about how Notes are used here, but I would assume that there would either be a listing of past notes or an editable ongoing note a la a wiki page? Is this the correct behavior?

Unrelated, but watching the wire, I also noticed that there are a bunch of CORS errors for some fonts and the like: This answers a question that I've often had about why sometimes my fonts are bold in topic titles and sometimes not: it seems that sometimes my browser is okay grabbing the fonts and sometimes it chucks them on the CORs error. Playing around a little, I can sometimes get the page fonts to load w/o CORs issues, sometimes with, but I'm not able to do it consistently yet. This could just be me though; I'm using FF 124.0b9 (64-bit) and Chrome Version 122.0.6261.111 (Official Build) (64-bit) on Ubuntu. It doesn't affect use or navigation at all so no worries, but I'm happy to try and make it reproducible if you're interested.

No bother at all. Hmm. You're supposed to have whatever text was in the box when you hit "save". There isn't any formatting, or list of notes -- it's just a big scratchpad. But it should definitely save whatever you paste in there (if you hit save, that is).

I'm not seeing anything obviously wrong with the notes code -- I was able to change my existing notes and save them, but that doesn't necessarily prove anything. Checking the database, I can see that you don't have a note entry at all. It's possible that it is only broken if you don't already have a record in the database, so my next step will be to log out and log in under a test account to see if I can reproduce the problem (and hopefully see something amiss in the logs). I'm definitely not seeing that behavior on my side, but I'm willing to believe it can happen. I'll have to re-familiarize myself with the way the fonts are set up, but in the mean time, it might be worth emptying your cache to see if you've just got something stuck.

So this was in fact broken. If you didn't already have a note record, it wouldn't create a new one. I suspect that this bug was introduced when I moved the notes out of the user record (they can be lengthy) and put it into its own table. Testing my own note functionality, it worked fine, but that's because I already had a note in the new table. Looking at the code, I suspect that it's never been able to create a new note since then.

And that's been (looks at watch) several years ago. And it's been broken the whole time. So, ummm... yeah.

I guess nobody is really using the Notes feature, which is fairly long in the tooth. If nothing else, this might be a reminder that the feature needs an overhaul.

In any case this bug is now fixed.

And for finding and reporting this bug, I bestow upon you the MV Entomologist Award. Wear it with pride.

Oh, addendum: the artwork for a bunch of the awards is likely to be changing soon. The rather primitive MV Entomologist Award artwork is going to be replaced by the image below (in smaller form, of course). You saw it here first.

I've been trying to make sense of this for the last hour or so, and I honestly am not sure what's going on. The fonts reside on a static (cached, CDNed) server, and the CDN has the most permissive possible CORS policy set. I'm not sure who would be throwing an error in this case, or under what circumstances, or why only occasionally. I suppose it's possible that one of the CDN edge locations is misconfigured (not under my control) but this seems unlikely.

Honestly, CORS makes my head hurt. Every time I try to understand it, or read an explanation of how it works, I end up screaming at the computer because the person that wrote the explanation doesn't understand that they are being ambiguous when they say "the server". WHICH server? The remote server? The host server?

So I'm going to set this aside for now. If you can reproduce it, and (better yet) tell me what makes it reproducible, I will be grateful.

Let me try and do my best to figure out what is going on, at least from my end. And thank you so much for taking the time to look into what may just be something local freaking out--CORS is no fun, especially when you can't see what's going wrong yourself.

I think I can reproduce it now, more or less. I'm using FF for the moment.

* In FF as above; new private window
* "Settings" -> "Privacy & Security"
* "Clear data..." -> "Cached Web Content" -> "Clear"
* Enter into bar https://modernvespa.com; forwards to https://modernvespa.com/forum/
* All fonts are received as normal
* Immediately refresh; all fonts throw CORS errors and do not load (causing the page to display all titles as bold)

Now, a fun wrinkle in this is that the above only gets the CORS errors (i.e. it /never/ gets the fonts) if I do it with the debugging window from the get go, implicating the browser at least partially some of the time. That said, I've found some version of these CORS errors in FF 123.0.1 (64-bit), FF 124.0b9 (64-bit), and Chrome Version 122.0.6261.111 (Official Build) (64-bit), so...hm.

It just seems to be the font assets. It looks like you're delivering them out of Cloudfront from and S3 bucket? An example (from my end anyways) "bad" asset is:

https://static.modernvespa.net/fonts/mulish/v2/mulish-v12-latin-ext_latin-700italic.woff

An example "good" asset is:

https://static.modernvespa.net/forum/v312/scripts/local_date.js

I don't suppose the two routes have different CF or S3 bucket settings or origins, do they?

Looking at the response headers, there are some differences, but nothing that seems like it should matter.

Grabbing curl (https://stackoverflow.com/questions/12173990/how-can-you-debug-a-cors-request-with-curl), I get the following kind of response for both the "good" and "bad" resource: Which is...confusing, as one of them works for me. The magic of CORS.

Again, it's fonts from the CSS here, so nothing that affects actually being able to use anything, but odd.

Thanks for this write-up. I'm pretty exhausted at the moment, so I'll have a look at this in the morning when I'm a bit more focused. You're correct about the architecture: The assets are in an S3 bucket, but CloudFront is between you and the bucket. I have the CloudFront distribution set to "Simple CORS Policy" (see below) as these are all just static assets -- fonts, images, CSS, and a few bits of javascript.

The two paths are in the same bucket, and use the same CloudFront distribution. I don't know of any substantive difference between the two paths, aside from the fact that /forum/vXXX/* changes regularly when images or CSS change (which just happened a few minutes ago, right as you were posting). I would say having the correct font is still pretty important.

Maybe it will make more sense in the morning.

It's been a lot of fun cruising around MV and looking at all of the updated award artwork, especially trying to locate unique awards given to various members. Very easter egg.

I would have bet that it was different bucket or distribution settings. (As a note to myself, that specific policy is https://us-east-1.console.aws.amazon.com/cloudfront/v4/home?region=us-east-1#/policies/responseHeaders/60669652-455b-4ae9-85a4-c4c02393f86c ). Looking at the stuff I've setup in S3/CF, we've had success with settings that look like yours, but also have a bunch of stuff that uses legacy settings and custom bits that all seem to work fine. Hm.

Taking a step back, I have another theory.

Looking at MDN for CORS simple request docs (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests), they have an interesting list of allowed and not-allowed headers. I'm wondering if I was too quick to dismiss the header differences between content-types.

My current theory is that it has nothing to do with the path or "font"ness, but rather that it's octet-stream content-type (which is the same set as the problematic files). It looks like there are additional headers and values being added that might make it inadmissible as a "simple request".

Later on, I'll take a look around to see if I can find a tool to verify that or not, as I don't trust my own reading of the spec.

(BTW, thank you for the My Notes fix--I can now happily edit and save!)

LATE EDIT:

I also noticed, looking at some other sites, that the mime-type is usually application/font-woff and application/font-woff2, instead of binary/octet-stream.

That's a worthwhile clue.

Following up on this, it looks like the correct mimetype (as of 2017) is font/woff and font/woff2

https://en.wikipedia.org/wiki/Web_Open_Font_Format#Browser_support

I'm going to try an experiment and change the mime type of an alternate set of the fonts in the static CDN.

besupa -- I've gone ahead and pointed the main site at /fonts/mulish/v3, which all have a new mime type. It seems to work correctly on my side (Safari, Chrome, and FF) but I haven't done anything more than a spot check.

Dunno if this fixes it, but it's at least more correct.

Thank you for continuing to look into this--I imagine it's a little frustrating. Unfortunately, a similar problem seems to persist. I wish I had documented it better, but I think the exact error may be a little different. I've added a screenshot now to make sure this issue is pinned down at my end.

The different behavior out of the same bucket/distribution is so weird to me; although it's very clearly talking about https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS/Errors/CORSMissingAllowOrigin , it seems to only apply to some assets...

Using the step-by-step you wrote up last night on FF 123.0.1 (on macOS) I am unable to reproduce the problem. Even with the browser console open.

Hmmm.

Nice! I think it's now sounding like 100% a personal problem, until proven otherwise

Thank you for all the checking at your end--at least you got some content-type updates out of it? I'll let you know if I can concretely figure out what's going on, and my apologies if it all ends up being a bit of a faff.

I'm definitely happy about making any improvement, and I'll do anything in my power to resolve your issue, if I can.

I did manage to get a screenshot of the results after my step-by-step:

Here's an example of something that makes me scream at the display. This is a very detailed description of some specific examples, but if you try to parse the actual relationship of components (particularly in the yellow-background section) it just flat-out does not make sense.

It's like all the explanations of CORS are deliberately trying to sabotage the functioning web.

besupa -- your specific use case sounds somewhat like this, though the particulars are different:

https://serverfault.com/questions/856904/chrome-s3-cloudfront-no-access-control-allow-origin-header-on-initial-xhr-req

EDIT: And this one too: https://stackoverflow.com/questions/31732533/s3-cors-always-send-vary-origin

jess, yes, that does feel close to what is going on (with the first one being a depressingly complex read). I think the thing I'm having the most trouble with is why it seems to occur for specific files and just in some cases. For fun...

...I just pulled out a Macbook (Sonoma 14.0) with FF 119.0, put it into troubleshooting mode, and still got the CORS errors...
...I updated to FF 123.0.1 and got them again...
...Using Safari (17.0) on the same Mac, no errors...
...Using Chrome on the Mac (122.0.6261.129), I can cause it to happen at will by using shift reload, and cause it to not happen (no errors), by hitting just reload...

Sooo...browser and cache related? Ugh. I'm going to go home and sprinkle my computers with holy water.

Yes, that's what I surmise. The issue seems to be that in some cases, browsers (mostly FF and Chrome) will read a resource in a traditional way that doesn't need CORS, but then use the Fetch API to get it a second time, but neglect to ask for new headers, because they are already cached. Unfortunately, the headers are not the same in the two different scenarios.

I've found various reports of this over and over again, in a time spanning almost a decade. The core issue seems to be that Chrome (in particular) disagrees with CloudFront on the correct behavior in this scenario, and so Chrome has opted not to fix what is very clearly an obvious bug that causes many people to tear their hair out.

One more reason why I do not use Chrome as my daily browser, but my personal preference won't solve this particular problem.

Here's another description of the same problem, though there are inconsistencies that I don't fully understand:

https://blog.keul.it/chrome-cors-issue-due-to-cache/

besupa -- it might be fixed. Maybe.

Or I might have made it worse. Maybe.

jess Well, hey now--I've been trying to reproduce for a few minutes now, but once caches were purged, have been unable to do so. At least from my end, things seem to be fixed! Thank you for this, especially as it was such a fiddly and hard-to-reproduce issue. I'll try some other browsers later today.

I'm very curious about the final fix. Would you mind explaining what the changes were that you made? Forcing CF to add certain headers?

w00t! Thanks for your help reproducing the problem. Yeah, exactly. I inferred that Chrome's core failure here is triggered by the fact that in some contexts CF/S3 returns the header "Vary: Origin" and sometimes it doesn't. I don't really understand why this is the case (which is a testament to how badly written the CORS spec is) but that's what I inferred from reading about a dozen different problem reports of very similar issues.

So I forced CF to add a "Vary: Origin" to all responses. I think I could have forced S3 to do it instead, but I'd prefer to leave CORS to CF and not S3.

I may yet have to tweak this, though. On Safari, when I check the response headers that were returned from the font, it is showing "Vary: Origin, Origin", which I take to mean that it is seeing it twice. This is not the case in Chrome or FF, though. More experimentation is necessary.

Lately, I am seeing a lot of traffic from undeclared crawlers, aka "bots". These are not the usual kind of crawlers (e.g. Google, Bing, etc) in that these bots are disguising themselves as regular users by publishing a standard user-like User-Agent string.

They're also not the usual spambots, which attempt to find a weakness in the defenses by continually trying elaborate sets of parameters designed to trigger a failure in the forum software.

Nor are they the usual high-speed-scrapers that breeze in and attempt to read hundreds or thousands of pages in under a minute. We have pretty good defenses against those (though not perfect) and usually stop them within about 10 requests.

This new breed of bots are fairly well-behaved, making requests at reasonable rates, and acting very much like the mature search engine crawlers (e.g. Google and Bing) -- except that they don't declare who they are, unlike mature search engine crawlers. In fact, these are almost certainly mature data-gathering entities of some kind, though it's not clear to what end.

Spot-checking IP addresses when I see them, it's a wide spectrum of possible culprits. Huwai-Cloud showed up earlier today, and had read hundreds or maybe thousands of pages from a single IP address, but paced itself enough not to hit the overlimit (high speed) threshold that would get it booted. I've seen a couple of cases where Comcast appeared to be scanning, but that is almost certainly someone doing something from a residential IP address. And then of course there's always plenty that resolve to well-known data centers (Azure, AWS, and even Google Cloud).

I'm okay with Google crawling the site, obviously. And even Bing. It's a mutually beneficial relationship. I allow a wide variety of other, smaller entities to crawl the site, even though the potential benefits are intangible. Pinterest? Sure. Why not. Apple? Well, I'm biased, so sure.

Then there are all the Fully Buzzword Compliant SEO Marketing companies that want to build their database and improve their SEO-juicing product, presumably so that they can try to sell it back to the site owners that provided the data to begin with. Screw those guys. I generally block them. At least they have the decency to declare their crawlers properly, though.

But the bots I'm really thinking about here are the stealthy ones, the ones that don't declare themselves but otherwise don't break any obvious rules. Though none of them are doing any tangible harm to the site, in aggregate they are a drain on resources. It's not going to break the bank, I don't think it's costing us a lot to serve these undeclared bots. But it's also not nothing.

Really, though, I think my real point is that Modern Vespa is for the humans, not for the bots. Allowing Google and Bing to crawl is also for the humans. Anything else is just an electronic vampire.

And rather than just block IP addresses of well-known data centers (which I have done on occasion, and am doing right now for Huwai), I think it's time to develop some heuristics to filter out the bots based on the fact that they don't behave like humans do.

I have some ideas. Let's see which of us is smarter -- the bots or the humans.

In all seriousness Jess, Thank you so much for all of your hard work keeping this place alive.

I, for one really appreciate it.

This is a very uncharacteristic response from the Bill Dog account.

I suspect a bot.

Goddammit

Reading through old threads, I noticed a former member's major award was no longer being displayed. The Molto Verboso award is displayed though. Not sure if this is a bug or a feature.

It's a feature. A lot of people were fairly peeved.

Really?
Nothing to see here, then.

I honestly don't remember how the discussion went down. Just that it was a sad day for everyone.

And here I was thinking y'all were just yukking' it up the moderators' lounge.
Sorry to hear it's not always a party- especially on this topic - I read it as a good-natured ribbing.

Bug alert: Is there such a thing as a locking curry hook? (Post 2671383)

Emoticons are not working in attachment titles

There's no BBCode processing on attachment titles. Emoticons are not expected to work there.