Modern Vespa

(Extremely tedious, geeks-only commentary follows. Most people will find this uninteresting. You've been warned.)

I have been in character set hell, on and off, for a couple months. A big database change I had intended was scuttled largely due to character set issues.

There are a bunch of related issues, but one of them is the very old software we started with, and the other is the relentless march of time. The original forum software (and database) used a simple latin character set (ISO-8859-1) which was completely reasonable at the time. It didn't handle extended unicode, but as the forum is primarily made up of English-speakers posting in English, it wasn't really an issue.

One of the problems that went unnoticed for a very long time, though, was the fact that Windows browsers, being authored largely by crazed chimps hopped up on adderall, started sneaking illegal characters into the forum database that were explicitly NOT part of the ISO-8859-1 character set. They were, instead, part of the almost-but-not-quite-identical CP-1252 character set. Both are 1-byte character sets, meaning all characters are encoded in a single byte. But the Windows CP-1252 character set defines a few extra characters in a range that ISO-8859-1 considers to be off-limits.

Most of the errant characters that made it into the MV database are so-called "smart quotes", by which I mean "@#$% Microsoft". There are a few other characters that show up as well, though: endash, emdash, ellipsis, and the Euro symbol. None of these have exact equivalents in ISO-8859-1, though reasonable substitutions can be made for most.

To be fair, the forum software should probably have rejected the incoming data encoded with CP-1252 characters. Even better would be if, every time a Windows user submitted a post with non-ISO-8859-1 characters, the forum would deliver a mild electric shock through the internet and into the user's hand through the mouse they clicked the "Submit" button with. Even better than that would be if, every time a Windows user submitted a post with non-ISO-8859-1 characters, the forum would cause a medium-sized turd to fall out of the sky onto the head of a random Microsoft programmer somewhere in the world.

But I digress.

After 16 years of accumulated cruft, I now have a fairly sizable mess to clean up. It's taken me a couple months to get around to actually fully understanding the magnitude of the problem, and while I feel like I've now got a pretty good handle on it, there remains the possibility that I haven't even scratched the surface.

I have just put in place a filter on all data coming into the forum that explicitly looks for the subset of CP-1252 characters that do not exist in ISO-8859-1, replacing some of these with close-enough versions. The Euro symbol is the most problematic, as there really aren't any good single-character substitutions in ISO-8859-1. In this case, the Euro symbol will be replaced with the word "Euro".

The long-term goal, though, is to NOT be limited to ISO-8859-1, but instead to allow the forum to accept input encoded as UTF-8. UTF-8 is widely regarded as the One True Character Set to support, as it largely eliminates all of the incompatibility by simply allowing every known character to be used. This means that you could, theoretically, post any character from any known language in the world, and the forum would happily accept it. And then Jim would unceremoniously delete that post on sight.

But I digress, again.

In order to get to the UTF-8 goal, I need the current MV database to be homogeneously ISO-8859-1 prior to conversion. That will mean writing new code to scrub every single post, every single post title, every single signature, and so on. I shudder to think how many usernames have random characters in them that don't conform to ISO-8859-1 -- those users may not be able to log in ever again. Hmmm. That would probably be bad.

I will then probably have to figure out how to re-index the search tables that MV uses for search. Or throw away the current search function and replace it with something better. Fun!

If you've made it this far through the technobabble, congratulations. I will leave you with one of my favorite jokes as a reward:

FTFY.

That just went Whoosh *hand flies over head*.
Looking forward to "Character Set Hell for Dummies".

I think my head just exploded.

My work here is done.

Is it sad that I read that and understood everything? I should say something witty and geeky here but I just got back from about 5 hours on the slab in rain, sleet and something else that stung like a bitch because the visor had to be up. Semis passing you in the pouring rain doing 70 mph is almost as much fun as character sets.

Nuke all the offending characters in posts and titles. Make a list of all the usernames with offending characters and ignore them for initial database conversion. Then maybe batch email those usernames to provide them with chance to change their name within set amount of time. Which then you can individually change their name and process their data into the new database.

Completely not the best way to do it. But sometimes a sledge hammer is called for!

UTFTW-8!

Indeed.

I feel your pain.

Once I finally honed-in on the fact that the CP-1252 character set was 90% of the problem, it was relatively straightforward to come up with a function to scrub out just that narrow band of characters. Previous attempts had resulted in lots of unintended consequences, like umlauts (and other legitimate ISO-8859-1 characters) turning into question marks, or disappearing entirely.

After much research on the interwebz, this is the function I came up with to scrub CP-1252 characters from any given string: I bet you're sorry you opened this thread by now.

I don't understand a dam thing in this thread, and don't want to as I have enough shit in my head already and having more will give me a head ache.

really I just like be numb and enjoy this forum most days for an hour here and there, this way I can not think about the other shit in my head.

I do appreciate the upgrades and faster speeds and the new bells and whistles. for those I am grateful.

So thank you for all your hard work in starting this forum and keeping it alive all these years.

Cheers to many more. wish I could find a drinking beer or whiskey Icon I can toast to you.

I'll put it on the list. No, wait, I'll do better: I'll post an emoji:

🍺🥃

I admit, I'm one of those who use the euro symbol but I didn't know I was doing something wrong. If you want I can not use the symbol but write the extended word or use an Eu.

No, it's fine. It's not the fault of the members who post. It's Microsoft's fault for submitting non-compliant characters to the site when the site clearly defines what character set is allowed. All part of their "embrace and extend" bullshit, which basically means "corrupt the open standards and make them unusable".

Ah ... you took a weight off my mind, I thought I fucked up ...

I could be wrong, but I think most of the time, when the Euro currency symbol is not available, it is replaced with EUR.

You don't mention it, but I guess you've heard that in MySQL, the utf8 character set is actually not true UTF8. Instead, you would have to use utf8mb4. Nice little trap there.

Broken umlauts is something we've been used to since the nineties. Here's a classic:
Die UTF-8 Selbsthilfegruppe trifft sich heute Abend im grÃ💱nen Saal.
The UTF-8 support group meeting tonight is in the green hall

and to your sir
🥃

The day that Microsoft makes a product that doesn't suck is the day they start making vacuum cleaners.

I recognize that quote. It's from a famous philosopher's musings on bing bot.

O.
M.
G.


{logs out-closes laptop-heads for the coffee maker}

Dear Jess,

I, also, have no idea what your post was about; your grammar, however, is excellent!
now, if I could just persuade you to indent your paragraphs.

You're not wrong. Standard currency name, like GBP and USD. would be far better.

Ah. Good point. I'll change that substitution. Yep. I'm aware. I stumbled on that as I was trying to upgrade the database to UTF-8 the last time, only to be foiled by the conversion process. That's an error called Mojibake. I've found some other examples of it on early posts at MV, but it peters out after the first couple of years. Apparently I must have changed some configuration at some point to diminish it.

I have this handy mojibake translator, too: https://www.i18nqa.com/debug/utf8-debug.html

That said, I can't find Ã💱 on the list -- was it intended to be ü?

Fortunately, even the ancient ISO-8859-1 can support umlauted characters, so there really should be no need to use UTF-8 for that.

Thanks, but... how do I put this as politely as possible? Indented paragraphs are a holdover from a bygone era. Like two spaces after a period, it is simply no longer the accepted standard.

¯\_(ツ)_/¯

That still works quite well!

My brain would have gone for escaped characters like &emdash;, but I suppose the forum doesn't allow them either.

Nice to hear you riding again Harbinger.

I understand just enough of this thread to know I'm glad it isn't me facing all that!

And I'll confess. I still mostly use two spaces at the end of a sentence. Old habits die really hard. I need to see if I can write an auto-hotkey to root that out. My spouse makes fun of me for it.

Back on topic - Thank You for all your work on MV!

I was taught the double-space convention in typing class. Just to put it in perspective, we were using manual typewriters.

I did have a nerdpost prepared about proper use of em dashes, but in the end just thought, 'nah, just post a shruggie'.

I'm not actually opposed to em dashes, and when the forum character set allows for it, I'll happily remove the restriction.

*sigh*
more evidence that I am old

I never indented except when required. However, within the last few months, I've (mostly) retrained myself not to double space after a period. Sometimes I falter but I'd been a 2-spacer since 1982.

What is the second half of the smile character?

Hey, I learned all this stuff once too. But in the intervening years, I've managed to unlearn those once-sacred teachings.

It's done. Well, the bulk of it, anyway. I've spent the last 48 hours or so writing code to convert various parts of the forum database, sanitizing and hand-fixing the things that didn't quite work right. There are still a handful of posts that need fixing, but I'm too tired to chase them down right now. I'll write another bit of code tomorrow to ferret out the stragglers.

I did have to change the usernames of about three users -- none who had logged into the forum in the last decade, though. So probably not a great loss. They can now use their email address as a login, though, so probably not a fatal change.

There are probably still lots of bits of illegal characters lurking in the darker corners of the database -- I won't know until I try and do a full export / import operation on the database. That will be... later.

Time for sleep.

Yes and no. It's a joke, so maybe that's not a real substitution. Also, when you see it on T-shirts and mouse mats and such, it's an uppercase A with tilde and a one quarter ligature, which I couldn't type on the iPad. So I picked the next best emoji.

Internet says the face part in the middle is a Japanese Katakana character.

ツ

... or Fiat cars ...

I, too, learned the double-space convention in typing class, on an IBM Selectric.

And in my accounting class, I was using one of two Osborne 1 portable computers the school had for the accounting class. Our computer classes used TRS-80 Model 3s, 4s, and 4Ps.

I still tend to double-space after sentences when typing. I know that HTML (theoretically) strips out the extra space, so I don't worry about it. And in iOS, the double-space is replaced by ". ", so it makes it easier. (I will admit that this post was composed with single-spaces at the end of every sentence, so I guess I'm slowly breaking the habit.)