Modern Vespa

(These are fairly arcane ruminations about obscure details that most people don’t know or care about -- you’ve been warned).

Many, many years ago, when Modern Vespa was still on its very first server (they just grow up so fast!) I was struggling to keep the server from overloading. MV was oddly successful right from the start, and we had more traffic than we could handle on such a low-end server. At the time, our hosting plan allowed us unlimited bandwidth, but a fairly stingy amount of compute time, with surcharges being assessed if we went over the monthly allotment. And we were going over the limit every single month. The bills were starting to add up.

One of the things I did at the time to alleviate the situation was to turn off gzip compression. (Sidebar: gzip compression is the standard mechanism to reduce the amount of data being sent across the wire between the host and the browser).

At the time, the compression was being done by the forum software itself, and it was definitely adding to the compute burden. How much it was contributing is an open question -- it’s trivial to compress text today, on modern hardware. It was a bit murkier in 2005/2006, when this was happening. Since I had unlimited bandwidth, though, and was being charged extra for compute, getting rid of gzip compression seemed like the right move. Do less work for every page served, even if it uses more bandwidth and even if the perceived response at the client will be slower.

Fast forward about 15 years, and the circumstances we now find ourselves in are the exact opposite. We are paying a fixed cost for our monthly compute resource, but we are paying by the gigabyte for bandwidth. The bandwidth charges aren’t exorbitant, but they’re not nothing, and they’re not fixed. So there’s an incentive to use less.

For the last 10 years or so, most of our non-bandwidth costs were quite a bit higher, and so the bandwidth charges were mostly just in the noise. However, over the last couple of months, I’ve been steadily chipping away at MV’s monthly hosting bill, finding better ways to provide the same service for less cost, and I’ve actually got the bill down substantially -- less than half of what it used to be. That makes the bandwidth charges stand out all the more, though. I’ve picked all the other low-hanging fruit, and bandwidth now represents the largest single item on the hosting bill.

And it turns out that in 15 years, I’ve never revisited the subject of gzip compression that I so unceremoniously ripped out of the software years ago. Lots of static resources (CSS, javascript, etc) are already sitting on the CDN server in a compressed form, but the HTML served by the core Modern Vespa server -- this topic, for instance -- has been delivered in uncompressed form this whole time.

I didn’t really think the HTML text was all that significant. As I started wondering what to do about the bandwidth costs, though, I looked at a few long threads to get some actual numbers. It turns out that a typical full page of postings is about 250k bytes worth of text -- that’s just HTML, not including avatars or attachments or anything. So it’s actually pretty significant. Well, okay, multiplied by thousands of page requests a day, it’s significant.

Fortunately, it turns out that this fruit was hanging lower than I thought, as it’s an easy problem to solve. Modern versions of Apache generally come pre-installed with mod_deflate, which is a module that will automatically compress page content on-the-fly if the client browser agrees to take it -- and all modern browsers do. Heck, even the ancient browsers accept gzip compression. It’s been a standard seemingly forever.

So I flicked the switched. And right off the bat, those large 250k pages started being more like 25k, or as little as 10% of the uncompressed size. Which means you all get your pages that much faster (especially when you’re on a cellular connection) and MV’s hosting bill will be a little bit less next month.

Win win.

I just wish I’d done it sooner.

Good on you for keeping the noggin's juices flowing.

Different times, different solutions, I guess. Glad you figured it out.

Will it survive Max’s picture posting habit though?

It is said that spinning the brain a lot by ruminating keeps brain atrophies away. I do puzzles and crosswords. Of that I am sure.

Looks like we're on track to save about $10 - $12 per month on bandwidth charges, just from this one change. Not bad.

Great!! This has always been the fiber's promise, lower transmission-cost/bit. It's somewhat unexpected (by me anyway) and interesting that fiber's transmission-cost/bit < the compression-cost/bit. A conclusion from this is that the advances in data transmission over fiber is advancing faster the processor speeds and energy usage. A very nice piece of work Jess.

Is it fair to ask the percentage reduction in monthly cost? Just curious. You can always PM me.

Miguel

It's fair to ask, but it's harder to give a satisfactory answer.

In order to lower costs, I've prepaid for the compute on two servers (the main server and a test instance) plus the database, so those don't show up on the monthly bill. But of course they're not zero, they're just pre-paid. What remains are charges for bandwidth and storage space (in various capacities) plus ancillary things like DNS hosting and email.

To give the true monthly cost, I'd have to work up a spreadsheet. But if you ignore the prepaid compute costs, our monthly bill is now down to about $60 - $70 per month. That's about half of what it was at the beginning of the year.

Oh, and bandwidth costs are still the largest category in the bill. Except now, the bulk of the bandwidth charges are coming from the edge-server CDN content, which includes all of the attachments, along with some static content like CSS and JavaScript files. I'm scratching my head trying to figure out how to make a dent in those costs.

The cost of actually shuttling HTML pages from the main server to your browsers should be around $2 this month, down from about $14 last month.

That is really significant percentage cost-reduction.Thanks. I appreciate the the info. I've intentionally avoided Internet Engineering over the years and focused on other technology areas, primarily transmission systems, so unfortunately, I don't have any suggestions for you on reducing costs in other areas.

All your efforts are much appreciated by all of us Jess. MV has been working great and I really like the new interface, speed and Karma feedback page. Thanks🌞👍🥂

Miguel

On the fibre/bit-cost front, I've just noticed my UK ISP has upped my monthly data allowance to 5TB per month for FTTC, plus half of what's left over from the last month, all at no extra charge. If you go over the allowance, speeds go down to ADSL levels with no extra penalty.

(FTTC - fibre to the cabinet, copper pair for the last few 100 yds or so. Speeds up to 80Mbps)

So you are using a different approach than Jess for MV. I'm guessing you own your own server housed in a data center, pay for rack-space with power, minimal support and connection to a backbone. In contrast, I think Jess pays Amazon and they handle everything, except his software. I'm guessing your costs will continue to decrease Jim because there's a low-cost fiber technology that is dramatically lowering the cost of fiber to the end user. It commonly called FTTX, where" X" is the application. FTTC is one, others include FTTH (home), FTTB (business), FTTM (mobile),... The underlying technology is based on passive optical networks (PON) that use completely passive components (no electrical power) except the end terminals. They have very low infrastructure cost (except the fiber laying) and they are much more reliable since they are completely passive (still can't withstand a backhoe tho) and are VERY cheap. They are being deployed in virtually every country. In the US, ATT and Verizon are offering FTTX. If you have fiber coming to your home from one of these carriers, it most likely PON. FTTX is also being targeted to provide the infrastructure for 5G cellular and that's a huge global market. There's much more to be said about all this but I'll leave it there.

Best
Miguel

No, this is a normal home setup, with a particularly competent ISP. https://www.aa.net.uk/broadband/home1/
https://www.ispreview.co.uk/review/top10.php

Jim, You have that computer housed in your CA home or in the UK? I have a bunch more questions about all that but I'd just embarrass myself with my relative ignorance so I'll hold. Besides housing repair manuals, what else are you doing with the server? Just curious. BTW, my professional focus is on broadband transmission systems for global telecoms, hence my interest in Jess' observation of bandwidth vs compression costs.

Miguel

Oh, the manuals are hosted with a separate provider. I pay GBP12 pm for that for all my domains (individual registration extra of course, currently 16 of them). No limit on data usage or data transfer amounts ('bandwidth') has ever been suggested. But all that was grandfathered in from the original "Gradwell Developer Account". Peter Gradwell then majored on VOIP systems, and retired from hosting.

Jess, I thought your observation about the shift between transmission-cost and compression-cost was suitably interesting that I shared it with some people I've worked with for decades.

A question came up: What is the compression ratio you got on average? I know you'll get different compression ratios between data and images but details are not critical. Just a general sense is fine.

Another guy said something I think you'll appreciate... Miguel

We aren't doing any compression on images, since they are already well compressed (by jpg or gif or png) and trying to compress them further generally just makes them larger.

Text compresses very well, though. We are getting anywhere between 6:1 to 10:1 compression ratios.

To be fair, though, the compute-vs-bandwidth tradeoff (in the case of MV, anyway) is muddied slightly. Amazon AWS has multiple "virtual machine" sizes and classes (you can see the full matrix here) to choose from. We are currently running on a t4g.small instance, but we're not using anywhere near the full capacity -- in fact, we're averaging below 5% at any given moment. But we've historically had some trouble with sudden floods of traffic, and so I'm wary of going much smaller. That said, this instance is performing way way better than our previous ancient instance (which was also in the small class) so maybe I've oversized the server somewhat. Live and learn.

But this means that for the next year or so, we are going to be paying a fixed cost for a server that has excess compute capacity, which makes the compression-vs-bandwidth decision obviously in favor of compression in this situation.

Even then, though, compressing text is pretty trivial these days, so I suspect that the compression-vs-bandwidth debate might be very well be permanently settled.

Some more details on bandwidth costs.

Last month, the main server pumped out 163.854gb of data (mostly HTML) at a cost of $0.09 per GB, for a total bandwidth cost of $14.75.

This month, we've done 23.065gb of data so far (with a few days left in the month) still at a cost of $0.09 per GB, for a bandwidth cost so far of $2.08.

There are other, more significant bandwidth costs (notably the static content, images, icons, javascript, etc) that don't reside on the main server, and so those costs are pretty much the same this month as last month.