Then, out of the blue, I'm getting reports that some tiles are not loading, which normally means tile generation bugs, but not this time. I look into the nginx logs and see this:

2025/08/08 23:08:16 [crit] 1084275#1084275: *161914910 open() "/mnt/ofm/planet-20250730_001001_pt/tiles/8/138/83.pbf" failed (24: Too many open files) ...

This is weird. I've never seen anything like this. I check nload, and it shows huge traffic. I log into Cloudflare and I see THIS for the last 24 hours.

What? 3 billion requests in 24 hours? What on Earth is that? Also, 215 TB of traffic from tiny, 70 kB files?

This much traffic would cost over $6 million per month on MapTiler and double that on Mapbox.

It's especially spiking in the last 5 minutes, where I'm seeing 30 million requests.

How much is that?

Wow, that's 100,000 requests per second!

My first thoughts are:

• Wow.

• Wow, I'm surprised that the only bug I've heard about is some missing tiles. I mean, the service still seems to work? Somehow those empty tiles got cached by Cloudflare, but other than that, it still kind of works?

Looking at the Cloudflare dashboard, I can see that 96% of the requests were 200 OK; only 3.6% were broken (206 Partial Content).

So it seems that OpenFreeMap is mostly managing to serve 100,000 requests per second?

But what is causing it?

Wplace.live happened. Out of the blue, a new collaborative drawing website appeared, built from scratch using OpenFreeMap.

I don't know what it is, but the internet seems to be crazy about it. One thing is for sure: the traffic it generates is out of this world.

I believe what is happening is that those images are being drawn by some scripts. If I understand correctly, the website limited everyone to 1 pixel per 30 seconds, so I guess everyone was just scripting Puppeteer/Chromium to start a new browser, click a pixel, and close the browser, possibly with IP address rotation, but maybe that wasn't even needed. When you create a beautiful an image by hand, you are literally spending half an hour or more over the same map area, you don’t make map tile requests during this time.

Nice idea, interesting project, next time please contact me before. Neal.fun did the same before launching Internet Roadtrip - they asked me if the traffic would be OK and even decided to sponsor the OpenFreeMap project with an amount that more than covers their bandwidth usage.

Sorry Wplace.live, I had to create my first-ever Cloudflare rule. As a single user, you broke the service for everyone.

I wonder if there is an automatic way to do this on Cloudflare, to limit traffic by referer or custom headers. It'd be great if this could be automatic, as I want to avoid this ever happening in the future. I hope I can script this via the API.

Finally, I want to say a huge thanks to Cloudflare for helping with the bandwidth. I haven't written about it yet, but when I contacted them back in November, they managed to get my bandwidth sponsorship status approved in 48 hours over a Saturday. Moreover, they connected me with some of their best engineers to discuss how OpenFreeMap could use their architecture even better. I've never seen a company of their size move as agilely as they did.

As the sole person running OpenFreeMap, I'm incredibly proud of these two numbers. First, that my architecture hit a 99.4% CDN cache rate, which is fantastic for a service with weekly data updates. And second, that my own servers successfully handled the remaining 1,000 requests per second.

Update

I managed to contact the dev behind Wplace.live. He said they grew to 2 million users in a few days, out of the blue, so I totally understand them not being prepared for this kind of traffic. BTW, I'm amazed that their main architecture is holding up.

I offered to help them set up a self-hosted OpenFreeMap instance, which is the perfect fit for such use cases. They get the service for free, the public instance doesn't get the load, and everyone is happy.

Also, "only" having 2 million users confirms my belief that it's mostly scrips creating the load. 3 billion requests / 2 million users is an average of 1,500 req/user. A normal user might make 10-20 requests when loading a map and editing over one place for a long time, so some of these are extremely high, scripted use cases.

Also, this is literally the 1st Google result below the official website.

Learnings

There are two learnings I'll write about in a next post.

1. I need to implement bandwidth limiting by referer. I'm looking into how to do this on Cloudflare. Nothing changes in the service; everything will still stay free and without registration, but every referer will be limited to a very high number, like 100 million requests per 24 hours or something similar. For native apps, I'll probably ask them to add a custom header identifying their app.

2. I'll need to improve my server config to fix those empty tiles. Even though I'm not expecting a load like this in the future, I think I found the culprit in the config that was causing the missing tiles.

If you find OpenFreeMap valuable, please consider sponsoring on GitHub. To be fully transparent, the project currently runs on $500/month in donations. This is just enough to cover all infrastructure costs, which is fantastic. However, it means that new development happens in my limited free time. More support means more time I can spend coding, ensuring OpenFreeMap is ready for whatever comes next.

You can sponsor the project on GitHub at https://github.com/sponsors/hyperknot

Right now, you might have read the news that a Chinese paraglider pilot, Peng Yujiang, got sucked into a cloud and ended up at 8,598 meters / 28,000 feet.

It's literally on every news website: CNN, BBC, The Guardian.

Now, what makes this story really interesting is the accompanying video. Here is one version from The Guardian, which had 300k views in 17 hours:

I've been paragliding for 18 years and follow the progress of AI very closely, but even I couldn't say with 100% certainty if this video is real or fake.

One scene is definitely fake: the one where the camera makes a move that would only be possible from a drone. In this scene, the otherwise black helmet is white, and the perspective falls apart in a strange, distorted way, typical of lower-quality AI-generated videos. Also, here, he is using a sitting harness (legs visible) whereas in the real ones, he is in a pod-style harness.

A screenshot from the fake scene:

I think the other two scenes might be real though, recorded with a 360 camera on a selfie stick - a common setup for paragliding videos.

There are definitely some questions:

• Why did he (or someone) have to make a fake AI-generated scene and insert it between the two real ones (if they are indeed real)?

• How does the camera move in both scenes, while his hands are definitely not moving it? In one scene, the camera makes a circular move, while in the other, it's moving away from the pilot slowly. I'm not saying it's impossible; I just don't yet understand what is moving the camera. Is the dolly-like move just a zoom effect? The resolution seems to remain the same.

• Why is the camera not frozen at all? Everything is covered in thick ice, but somehow the camera's lens is not. (The phone's screen is also perfectly clear and ice-free.)

  

• To me, it seems like the pilot's face slightly changes between the two scenes. In one, he is a bit skinnier and paler; in the other, he is a bit more rounded and darker in color. Maybe the two scenes were recorded at different times?

Looking closer at the equipment, we can conclude that this is indeed the case. He is using a different harness with a different setup in the two scenes.

In one scene he has a phone with a glossy screen (left arrow), whereas in the second image, he has a plastic power bank.

Also, the harness is different: one has shiny metal connectors (right arrow), the other has orange straps.

So we can definitely conclude that these two scenes were recorded at different times.

What is the origin of this video?

Here is a YouTube search page for "Chinese paraglider" - ordered by upload date.

The oldest version is from 4 days ago, interestingly from the possible manufacturer of the hand-warming "mittens" he is using. Maybe the whole idea came from the manufacturer?

After that, there is this YouTube Short, which, to me, looks like the best-quality footage so far.

Afterwards, AI-generated fake scenes start to be mixed into the videos, which in turn are copied by all major news outlets.

Even the No. 1 paragliding magazine is linking to one such video, whereas it should be obvious to every paraglider pilot that it contains AI-generated scenes.

Don't worry too much about the articles saying he just accidentally got "caught" up in the cloud; this was a super-well-planned and carefully executed flight. He, of course, has to say this; otherwise, he'd get in even bigger trouble with the authorities.

Additionally, it appears that this is a mechanical altimeter. Paraglider pilots typically never use such instruments because our electronic devices are much more precise, more lightweight and function reliably at the altitudes and temperatures we normally encounter. This particular gauge is a bulky, industrial-style metal instrument, specially carried for these record attempts.

How is it possible to do this? Wouldn't someone lose consciousness before reaching those altitudes?

Yes, most people would lose consciousness long before reaching such altitudes. However, Sherpas can climb Mount Everest without supplemental oxygen, even while performing intense physical activity. Therefore, I don't see why a suitably adapted person couldn't reach the same altitude, comfortably seated in a paragliding harness.

So, to answer some of my questions, I believe:

• The video contains three scenes: one is 100% AI-generated, one is from an older attempt, and one is from the current flight.

• The dolly effect in the current flight might be AI-generated, but the footage is real. The linked YouTube Short is definitely original footage.

• I believe the pilot definitely made the flight, and it was planned long in advance, possibly over many months or even years. He might even have had a sponsor from the mitten manufacturer.

• Interestingly, in that climate, all glass surfaces stayed clear of frost, so the camera had a perfect picture. I mean, airplane windows also don't get icy from the outside, so it's possible that glass just stays 100% clear in that environment.

• It's a pity that an AI-generated scene got mixed into the video, as well as an older recording. If not for this, the footage would have been authentic.

The real story, however, might be how major media outlets shared this as authentic footage, yet the AI-generated portion looks laughably fake by today's standards.

But here's what's truly concerning: what we consider state-of-the-art AI generation today will look just as crude in twelve months. AI video quality makes massive leaps every year. If news outlets already struggle with primitive AI content, how will they cope when the technology becomes perfect?

For finish, below are some videos possible with today’s latest model, Google’s Veo 3.

Bonus: current state-of-the-art AI video samples

Emotional support kangaroo not allowed on airplane

Prompt theory

By @emollick

By @PJaccetturo

In my experience, package.json was a stable, proven system that worked. Then came corepack's introduction of adding packageManager in each package.json, even when no one asked for it, for example when simply running “pnpm i”.

The idea is that to enforce reproducibility, you have to pin not just your packages, but also your package manager - and not just to a major version, but right down to the patch version.

In a kind of micro-management style, node/corepack now takes control from the package managers. Instead of trusting them not to break reproducibility, they introduce "phantom" package managers based on which directory you're in.

So for example, if I run:

$HOME/.nvm/versions/node/v20.17.0/bin/pnpm --version

it reports "9.10.0" when I'm in one folder and "9.15.2" when I'm in another directory. This totally breaks how "bin" files are supposed to work on UNIX/Linux/macOS. One "bin" file should equal one version.

So what’s in the file? It seems to be a wrapper calling corepack.cjs:

#!/usr/bin/env node
process.env.COREPACK_ENABLE_DOWNLOAD_PROMPT??='1'
require('./lib/corepack.cjs').runMain(['pnpm', ...process.argv.slice(2)]);

Unfortunately corepack.cjs is a unreadable/bundled file, so it’s not easy to figure out what’s happening inside.

They added a possible way to turn this behaviour off (set COREPACK_ENABLE_AUTO_PIN to 0), but hid it in an ENV var, so it cannot be set in a repo - it has to be set on each contributor's machine. The first Google result for this COREPACK_ENABLE_AUTO_PIN env var is a GitHub issue, explaining why the default setting breaks open-source workflows, as each contributor is required to disable it in their local environment - it cannot be configured on repo level.

Moreover it's broken. Every time I run “pnpm i”, it annoys me to upgrade pnpm:

Except, when I run the displayed command:

corepack install -g pnpm@9.15.2

it does nothing. It definitely doesn't update the packageManager field in my package.json, so my project is always left on the old version. This means that tomorrow, it can tell me again that "Update is available!"

The confusing thing is that the update notice links to pnpm’s GitHub and even to their X page. There's not a single mention that its corepack which is artificially keeping back the package manager's version.

I opened a thread on pnpm where they recommended me to turn off this behavior of corepack. Which is basically the only choice they have, as corepack is breaking this system above their head and then users report thinking that pnpm is broken.

GitHub links

Open source maintainers explaining why it's not working (29 upvotes) https://github.com/nodejs/corepack/issues/485

My issue for pnpm trying to figure out what's happening: https://github.com/orgs/pnpm/discussions/8911

My issue for corepack: https://github.com/nodejs/corepack/issues/587

Cost of AI models:

• Training time: a one-time, 3-month-long process costing up to a billion dollars

• Inference time: when you ask questions in your favorite chat app. The cost of this is converging to a fraction of a cent.

The second stage of the AI age is all about your data

Ilya Sutskever announced in a NeurIPS talk, that AI labs are running out of data, "The fossil fuel of AI".

"We have but one internet" - the data on the internet is limited (also meaning that they've already scraped whatever they could).

In light of this, some of the recent announcements by the companies behind the AI labs make immediate sense.

Microsoft is offering Copilot for free, previously $20 per month (still a loss leader even at that price!). The catch: in all plans under $19 per month, they are training on your data by default.

Google releases Gemini 2.0 Flash, one of the most advanced models ever, with real-time audio and video-streaming input! And if that's not enough, you get it entirely free on Google AI Studio and through the API!

The catch? When you use the Free version, Google uses your data for model training. At least they clearly point out that you should not submit "sensitive, confidential, or personal information to the Unpaid Services".

(Also, can anyone point out how to access the non-Free version? I really tried but couldn't find how. Do you need a Google Cloud account for that?)

OpenAI is offering previous premium features, like Search, to all users, even those on the Free plan. They really want to see Google dance. (Meanwhile, Google makes them dance with the Veo 2 video model - it's really fun to watch.)

They also now give Free users a monthly allowance of "Advanced voice mode" and have even started 1-800-ChatGPT, a free phone line where you can talk with ChatGPT!

The catch? You guessed it - for all consumer plans (aka not Teams or Enterprise), your data is used for model training. Ironically, this seems to include even the new $200/month Pro plan!

One company surprisingly missing from the last 2 weeks' series of announcements was Anthropic, which happens to be the only company that is not training on your data.

They seem to be taking a different direction compared to other players, and they also happen to have the best model for coding! Claude Sonnet is actually so good that Microsoft was forced to start offering it in GitHub Copilot; otherwise, developers would have left the platform.

It's wonderful to watch a dance of titans, who are outbidding each other while offering loss-making products for free. Just remember to stay aware of how they handle your data and which providers/plans actually respect your privacy.

Also, to be entirely honest, there might be other reasons for offering these products for free. For example, by offering CoPilot for free, Microsoft overnight decided the fate of dozens of "coding assistance" startups. The chances of a new "Cursor" being started in 2025 just got quite a bit slimmer.

I’ve also written 3 TIL posts since the last newsletter.

Here are the shortest commands I could come up with for eslint and biome:

eslint

eslint --debug 2>&1 | grep /repo | sed 's|.*/repo||' | sort -u

biome

biome check --colors=off --log-level=info --log-kind=pretty . | grep 'path' | sed 's|.*\./||' | sort -u

update: biome recently added file listing to the verbose flag, so it can be simplified to:

biome check --verbose

For Python, ruff supports a flag (--show-files), but only for "check", not for the "format" command:

ruff

ruff check --fix --show-files . // works
ruff format --show-files . // doesn’t work

Why do these tools make it so difficult to see this information? Prettier shows it by default on every run - not even a flag is needed.

From what I've seen, there's one universal library from Vercel called AI SDK. It has 870k weekly downloads on NPM and a fancy package name: just "ai" (I wonder how much that one cost).

It's split into 2 packages, and the "Core" one sounds promising! So how much does this "Core" add to the bundle when using it for just a single OpenAI request?

A whopping 186 kB! For a tool that simply calls an OpenAI endpoint. What is basically just a JSON request.

I really don't get Vercel and the whole ecosystem they built with VC money. Everything they do is the most overcomplicated, architecture-astronaut level stuff. Like here, just the "packages" subdir is 73 thousand lines of code. For a JSON API wrapper!

I mean, the "bad guys" story would be: let's create extremely bloated JS libs so that you'll pick our solutions which allow you to run these on our servers. But I really don't think this is the case - it's just that Vercel's company culture is all about overcomplicating everything.

I've read from many devs that you choose Vercel for the amazing DX, but I just really don't see that DX. Luckily, I haven't had to refactor Next.js codebases every year, but even from what I've seen, everything they do is the opposite of great DX.

Anyway, back to my original quest - I haven't found any minimalistic, universal JS lib that supports streaming from multiple providers, so I'll be writing it myself using the low-level "eventsource-parser" lib.

Has anyone found anything for this purpose?

1. Inline styling is really painful to type in SolidJS, as it uses "margin-bottom" instead of marginBottom. You need to type the quotes and dashes, and there's no autocomplete. Meanwhile, your competitor wrote "mb-2" in Tailwind.

2. Tailwind has the highest performance, with absolutely no runtime, but I just prefer real CSS.

3. CSS modules doesn't allow me to write CSS next to my components. For tiny components/helper utilities, I don't want to write two files. Also, CSS modules is runtime only, meaning slower performance.

4. vanilla-extract is like CSS modules but compile-time and super fast. Still, it requires writing everything in separate files.

5. Then comes Panda CSS - I lost a day trying to make it work. It has perfect SolidJS support, but it's the single most overcomplicated CSS solution I've ever seen in my life. It's like Kubernetes for CSS. Moreover, simply adding it to a project totally overtakes its CSS into a "design system" and introduces 14 kB of minified CSS! It also adds a whopping 51 kB of runtime JS! For a library that's supposed to be a compile-time CSS solution, this is crazy.

I chose it because I like Ark UI and it has a matching Panda styled "Park UI", but I just ended up having really bad feelings about how Panda would work in a long-term project. What I wanted was a simple CSS-in-JS solution for setting styles on components; what I got was a super complicated system that tells me how to do everything in their "design system".

Case in point, removing the border around the Accordion component from Park UI was so complicated that I ended up asking about it on their Discord. The "copy your component to simply modify it" approach is totally broken here. Any other solution (Radix UI, shadcn, SolidUI, etc.) would have allowed me to tweak the rule in no time.

6. "macaron" is recommended on the SolidJS docs site, but it's done by a single person, Google knows nothing about it, and it doesn't even work.

7. Solid Styled Components and Solid Styled JSX have no commits in 2 years. Also, that Styled JSX syntax is really not pretty. Both of them are runtime only too.

8. Pigment CSS/linaria doesn't support SolidJS.

9. In the end, I went with styleX by Facebook. It's minimalistic, unlike Panda, allows using the same file for JSX and CSS, and is actually extremely fast! Its full runtime code is only 6 kB, and you can see from the quality of the repo that Facebook has been using this in production for 5+ years. Also, it works with SolidJS, which is a surprise from the creators of React.

10. So styleX it is - I'm very happy with the minimal approach it takes. I only wanted to type some styles in JS, and that's exactly what it allows. It doesn't want to introduce a "design system", doesn't want to change the way I write code; it just stays in the background and generates atomic CSS classes at compile time.

Background 1: 1.5 years ago I wrote an article about it.

Background 2: Back then I invested quite some time into using Next.js and modern React, but soon after I wrote the article, this Next.js + modern React combo became the most disillusioning experience in my software developer career. I think I really don't get Vercel; it's supposed to be amazing DX for a lot of money, but I honestly found it the worst and most confusing overcomplicated DX for a lot of money. I think Vercel is single-handedly the reason for the renaissance of HTMX / FastHTML / Hotwire / LiveView frameworks. Which is actually quite good for the ecosystem!

Background 3: I'm trying to figure out how I can offer Enterprise SSO for $300 per month plans on a SaaS product, which is 10-20 times smaller than where it's usually offered.

What would I use today?

1. A separate backend API in any language the developer prefers. For example, for AI projects, writing the backend in Python makes a lot of sense.

2. Splitting the frontend into app subdomain and home page / "marketing" website. Marketing website can be a static site in a CloudFlare bucket. I really like Astro for this part. The tricky part is the pricing page, which can depend on local currency, IP address, etc.

3. The app lives on its subdomain, like app.myproject.com, which requires login. This one can be a classic SPA created with Vite. No need to overcomplicate anything, it's a breath of fresh air to see a classic SPA after battling Next.js's bizarre server/client magic.

One bonus advantage of this system is that Google only sees the marketing website, so we don't get penalized on SEO for any technical or UX choice we make on the app subdomain.

Now, what happened on the auth landscape in the last 1.5 years? First of all, there are so many auth providers now that it's impossible to count them all. Michael Grinich, the CEO of WorkOS listed 31 on the HN discussion of my article here, but even then he missed a few.

Almost all the providers I wrote about in my article changed pricing. I don't mean just a little bit, but they totally rethought their pricing structure/logic. It is probably my biggest worry about recommending closed-source vendors, that with auth you are at the ultimate vendor lock-in and the vendor can just decide to refactor their pricing model any time.

On the positive side, many limitations I wrote about in that article got fixed. For example, Supabase added 2FA and fixed some bugs, ZITADEL added nice examples and worked a bit on the UI.

Ory totally moved up-market and is now expensive and open-core. Price for 10k users is $1600 per month. Self-hosting is not an option anymore, as the open-source version doesn't support SAML orgs for example.

Also, they have some really weird hard-coded behavior, like when changing email address, you need to verify both the old and the new email address. In my life, I've experienced this behavior maybe once, and that was on a bank account. Literally no SaaS vendor asks you to verify your old email address. The reason you want to change it in the first place is probably because you no longer have access to your old email!

All in all, I no longer recommend Ory.

Clerk is super focused on the JS ecosystem, especially Next.js. It's lacking SDKs and examples for other languages.

Moreover, they ask $125 for MFA, which I find super baffling; it reminds me of those web hosts who asked for money to have HTTPS even many years after Let's Encrypt existed. MFA is one of the reasons people go with dedicated auth providers, don't sell it for $125!

I no longer recommend Clerk.

FusionAuth is closed-source and they offer a limited version for self-hosting. Their pricing is super confusing.

To get the full package, like SCIM, it costs $3300 per month for 10k users and you even need to self-host it!

BoxyHQ offers a really interesting open-core model, with the super geekily named SAML Jackson (it takes a while, try to say it out loud).

I found their product to be the most promising for adding SCIM or Enterprise SSO onto an existing auth setup. It's not yet clear how these combined products work. I mean they offer the SSO login while you are already using some kind of auth system, how do you handle sessions? Do you code two, independent session handling mechanisms into your app? That sounds super bug prone.

If not, then how do you combine your existing auth provider with BoxyHQ? I see that they offer some sample apps with Supertokens for example, but I don't yet understand the general logic here. If you know how these mix-of-two-provider auth systems work, can you explain it? It seems very bug prone to me.

WorkOS is super good! They are a serious company, which used to offer only Enterprise-related features, and recently started offering "normal" auth, like password or social login.

They have the easiest to use website and examples in many languages. It's still early stage for their product though; for example, changing emails is still a work-in-progress. I wouldn't worry about it too much, as if you start developing a product now, then by the time you ship and get traction this feature is probably shipped.

Also, the best part is that they offer 1 million monthly active users for free! Basically their pricing model is you get everything you need for B2C websites for free, and you pay premium money for enterprise SSO / SAML / SCIM.

With that said, I arrived at their biggest limitation: the enterprise SSO with directory sync / SCIM costs $250 per month per customer. At which point you have to ask the question how much "auth tax" can you afford per customer? If the answer is say 5%, then you need $5000+ per monthly plans. Which is a very realistic scenario for their bigger customers, but definitely not for all companies.

Having said that, if you don't need SSO / SCIM then I think there is almost no question, the best solution is to use WorkOS! I mean it's the perfect choice for 99% of B2C apps, and is totally free!

And what if you do need those features? For example, what can you do if your customers require enterprise SSO / SAML on a $300 plan? You look for cheaper offers or self-host.

Supabase is one of those cheaper offers, offering SAML on their $25 Pro plan! What's confusing is this page in their docs, which says that you need the $600 per month plan.

Also, it's not clear if any of these advanced features are available in the self-hosted version, or it's tied to their hosted offerings.

Right now I'm checking out self-hosting ZITADEL.

Previously I wrote that the configuration is very complicated, which is still true today, but luckily they added nice documentation, example projects and even videos about how to set up auth for the most common scenarios. For example React frontend + Python backend is one of their examples!

I'll write about ZITADEL as I work through it. They are one of the few truly open-source companies in the auth space, and one of the very few solutions offering open-source Enterprise SSO / SAML! Their whole product is open-source and is shipped in a Docker container, so it's relatively easy to self-host!

But first, a comment about their pricing for the hosted version. They give the pricing on Daily Active Users (DAU), which is calculated in the weirdest way. It's so weird that their first 2 FAQ items are trying to explain it! Basically it's saying that if a single user checks your site every day, then it counts as 30 daily active users. It sounds like the craziest thing to me. For me this seems like 1 user, definitely not 30. Is this normal to count daily users like this elsewhere?

Also, doesn't it just push developers to set a really long refresh timeout, as users are counted when a refresh token is issued? I mean, if they are pushing developers to use less secure environments, then what's the point of choosing a dedicated auth provider in the first place? Isn't the whole point of security is to have reasonable duration refresh tokens, which can be revoked if needed?

Anyway, they really have the best open-source offering and if you self-host then the pricing totally doesn't matter!

That's what I've learned in the last 2 days. If I wrote something that isn't correct, or if you can help me with some of my questions, or if you just feel like commenting, please leave a comment below.

What is Round Robin DNS?

Normally, when you are serving a website from a VPS like Digital Ocean or Hetzner, you add a single A record in your DNS provider's control panel.

This means that rr-direct.hyperknot.com will serve data from 5.223.46.55.

In Round Robin DNS, you specify multiple servers for the same subdomain, like this.

This allows you to share the load between multiple servers, as well as to automatically detect which servers are offline and choose the online ones.

It's an amazingly simple and elegant solution that avoids using Load Balancers. It's also free, and you can do it on any DNS provider, whereas Load Balancing solutions can get really expensive (even on Cloudflare, which is otherwise very reasonably priced).

How does it work in theory?

I became obsessed with how it works. I mean, on the surface, everything is elegant, but how does a browser decide which server to connect to?

In theory, there is an RFC 8305 called Happy Eyeballs (also linking to RFC 6724) about how the client should sort addresses before connecting.

Now, this is definitely above my experience level, but this section seems like the closest to answering my question:

If the client is stateful and has a history of expected round-trip times (RTTs) for the routes to access each address, it SHOULD add a Destination Address Selection rule between rules 8 and 9 that prefers addresses with lower RTTs.

So in my understanding, it's basically:

• Checking if servers are online/offline

• Sort the online ones according to ping times

In Practice

Let's see how it works in practice.

I created 3 VPSs around the world: one in the US, one in the EU, and one in Singapore. I made 3 proxied and 3 non-proxied A records in Cloudflare.

They run nginx with a config like this:

server {
    server_name rr-direct.hyperknot.com rr-cf.hyperknot.com;

    # this is basically a wildcard block
    # so /a/b/c will return the same color.png file
    location / {
        root /data;
        rewrite ^ /color.png break;
    }

    location /server {
        alias /etc/hostname;
        default_type text/plain;
    }
}

So they serve a color.png, which is a 1px red/green/blue PNG file, as well as their hostname, which is test-eu/us/sg.

Client Behavior When Servers Are Online

I made a HTML test page, which fills a 10x10 grid with random images.

The server colors are the following:

• US - green

• EU - blue

• SG - red

Important: I'm testing from Europe; the EU server is much closer to me than US or especially the SG one. I should be seeing blue boxes!

Chrome

Chrome selects somewhat randomly between all locations, but once selected, it sticks with it. It re-evaluates the selection after a few hours. In this case, it was stuck with SG for hours, even though it is by far the slowest server for me.

Also, an interesting behavior on Chrome was when not using HTTP/2: it can sometimes choose randomly between two servers, creating a pattern like this. Here it's choosing between EU and US randomly.

Firefox

Behaves similarly to Chrome, selects a location randomly on startup and then sticks with it. If I restart the browser, it picks a different random location.

Safari

To my biggest surprise, Safari always selects the closest server correctly. Even if it goes offline for a while, after a few refreshes, it always finds the EU server again!

curl

Curl also works correctly. First time it might not, but if you run the command twice, it always corrects to the nearest server.

If you have multiple VPSs around the world, try this command via ssh and see which one gets selected:

curl https://rr-direct.hyperknot.com/server
test-us

curl https://rr-direct.hyperknot.com/server
test-eu

Cloudflare

Cloudflare picks a random location based on your client IP and then sticks with it. (It behaves like client_ip_hash modulo server_num or similar.)

As you have seen in the images, the right-side rectangle is always green. On my home IP, no matter what I do, Cloudflare goes to the US server. Curl shows the same.

curl https://rr-cf.hyperknot.com/server
test-us

Now, if I go to my mobile hotspot, it always connects to the EU server.

If I log into some VPSes and run the same curl command, I can see this behavior across the world. Every VPS gets connected to a totally random location around the world, but always the same.

curl https://rr-cf.hyperknot.com/server
test-sg

Client Behavior with Partially Offline Servers

So what happens when one of the servers is offline? Say I stop the US server:

service nginx stop

Chrome

Firefox

Safari

curl

curl https://rr-direct.hyperknot.com/server
test-eu

As you can see, all clients correctly detect it and choose an alternative server.

Actually, they do this fallback so well that if I turn off the server while they are loading, they correct within < 1 sec! Here is an animation of the 50x50 version of the same grid, on Safari:

Cloudflare

And what about Cloudflare? As you can see in the screenshots above, Cloudflare does not detect an offline server. It keeps accessing the server it decided for your IP, regardless of whether it's online or offline.

If the server is offline, you are served offline. In curl, it returns:

curl https://rr-cf.hyperknot.com/server
error code: 521

I've been trying to understand what this behavior is, and I highly suspect it's a bug in their network. One reference I found in their documentation is this part:

Based on this documentation - and by common sense as well - I believe Cloudflare should also behave like browsers and curl do.

Cloudflare Wish List

1. At the very least, offline servers should be detected.

2. Moreover, it would also be really nice if the server with the lowest latency were selected, like in Safari!

I mean, currently, if you have one server in the US and one in New Zealand, exactly 50% of your US users will be served from New Zealand, which makes no sense. Also, for Safari users, it's actually slower compared to not using Cloudflare!

There is a HN discussion now, where both the CEO and the CTO of Cloudflare replied!

Note 1: I've tried my best to understand articles 1, 2, 3 which Matthew Prince pointed out to me on X. As I understand, they are referring to Cloudflare servers as "servers", not users' servers behind A records. Also, I couldn't find anything related to Round Robin DNS.

Note 2: If you have any idea how I can keep running this experiment without paying for 3 VPS-es around the world, please comment below. I'd love to keep it running. Is there a serverless platform that supports HTTPS and Round Robin DNS?

I launched OpenFreeMap on HN exactly a month ago. Well, it's not exactly correct, as I first submitted it to HN 5 days before, when absolutely nothing happened (1). The second time it became the 12th most successful Show HN of 2024 (until today), with 848 points!

What did I change during those 5 days? I went on Reddit and launched it on r/selfhosted, r/opensource, and r/webdev, each time iterating on the title and the description.

The great thing about Reddit was that it allowed me to interact with the community; some users even started a chat. It helped me realize what the important values of my project were - the self-hosting and open-source nature - while the free part isn't really special.

Which is strange, as the technically challenging part of this project is to offer free tile-hosting for public usage, but I guess developers are fed up with "free" projects - and I totally understand them.

Here are the before & after titles of the two submissions.

Reception

I was worried about launching an open-source project, and I was especially worried about HN, a famous place where projects are torn to pieces and criticized harshly. To my biggest surprise, I got the total opposite on both Reddit and HN: very supportive and helpful comments!

Seriously, I was very humbled by how nice everyone was; I think it was one of the most positive Show HN threads I've read. The comments about the support plans/monetization were so on point that I decided to remove the whole section about support plans and simply choose GitHub Sponsors as the sole donation source.

On GitHub, the project got 2k stars in less than 2 weeks!

GitHub issues/discussions were about the following topics:

• Issues about bugs related to self-hosting and special setups - I believe I have fixed all of them by now. Thank you all for reporting them!

• Questions which I thought were obvious, but it turns out they definitely are not. For example, the fact that the map on the homepage has scroll and rotation disabled doesn't mean that you would have it disabled on your website. I need to explain this more in the documentation.

• And finally, there were some questions which were not directly related to OpenFreeMap, for example, transliteration issues in Planetiler.

I also got the first successful reports of self-hosting working, and one user even pointed out an amazing €4.5/month VPS from Contabo which is perfectly capable of running this project. I'm super happy to hear that, as it might be a key element for the future of this project.

Community contributions

CJ created a fantastic video on YouTube about this project. It's a great resource that contains a lot of information about the background of online maps and even has code examples in five different frameworks.

Traffic

In the last few days, traffic has started sky-rocketing. That's about 120 TB/month with the current speed on 2 servers.

I looked into it and discovered that Sony Music launched a project using OpenFreeMap; almost all the traffic comes from this single website.

It's actually quite a nice website; I like it. Technically, it has no reason to load the full planet map. This could be just a 500 kB GeoJSON file, but the world isn't full of GIS experts who know how to make a simplified GeoJSON file with countries and US states merged. (Hint if you want to do this: go to https://geojson.xyz/)

The realistic alternative would have been for them to go with Mapbox or Maptiler.

So how much would this cost on those platforms?

This single website is making about 300 requests per second. That's about 777 million requests per month.

On Mapbox, it would be about 200 million "map loads." With a discount of $2 per 1,000 loads, it would be $400,000 per month.

On Maptiler, it would be $300,000 per month.

I'm actually really happy this works. If OpenFreeMap can serve $300,000 worth of map data on infrastructure which costs $175 per month, then I think it really has some value!

The Future and Financials

During this month, I have been thinking a lot about the future of this project, especially the financial part.

There are 3 parts:

1. Infrastructure cost

2. Maintenance work

3. Improvements and new features

The Server Cost

Currently, the server cost is about $175 per month, and the GitHub sponsors are at $105 per month. The first milestone would be to reach $175; then the project would cover the server cost! Please sponsor if you can!

This should be good for about 300 TB per month on Hetzner.

Of course, if Sony launches a few more projects on OpenFreeMap, then I will need to add more servers or figure out a way to limit the load. Some ideas:

1. Make a really simple-to-follow self-hosting guide for Contabo VPS. Everyone can set up their local instance for 4.5 EUR per month.

2. Make a guide for proxying and adding a per-user CDN.

3. Reach out to Bunny or Cloudflare for sponsoring a CDN.

4. As a last resort, limit extreme load coming from a single website, like over 1 million requests per day. This really is a last resort.

The Maintenance

The other part of the future is maintaining the project. This one has quite a black-and-white answer: I need to do this anyway. OpenFreeMap is the tile server for MapHub, and MapHub is my main income.

Basically, if I have to do it anyway, why not do it in the open?

Also, I'll remove the SLA part from the website. I want as good an SLA as possible; there is no point in differentiating and making two separate infrastructures, one with "free" load and one with "premium" load. The free service should have as good an SLA as possible.

Improvements

The third part of the future is improvements. For example, making new styles, fixing icons, etc. I believe this part can rightfully be connected to the financial success of the project. That is, priority #1 is to cover server costs, and if donations reach that, then I can invest time into improvements.

I'll work on the homepage to reflect these thoughts; I believe this direction could actually work.

If you have ideas about sponsorship, like which projects/foundations should I reach out to, please contact me.

(1) Well, not exactly true. @kiwicopple, the CEO of Supabase, noticed it and became my first GitHub Sponsor. Thank you!

For years, everyone understood they had to click the "Save" button to preserve their changes. However, recently, I've been receiving more and more support requests along the lines of:
"I lost all my work" or
"I've been working on a map for days, and now I can't find it."

I explained to everyone that there's no need to worry; even if they accidentally edit or delete their maps, they can always recover previous versions or even their deleted maps by clicking on the "Version History" or "Restore Deleted Maps" buttons.

But more and more users were telling me they couldn't find anything. That's when I realized that many users today simply aren't used to saving files manually.

They've grown up using cloud-based editors like Google Docs, where autosave is the default. Of course, autosave is only possible if you have a robust, automatic history-tracking system built into your app, which Google Docs certainly has.

The thing is, you can only offer this feature if your app's architecture is designed from the ground up to support it. Since MapHub wasn't built that way, I had to come up with a solution to remind users that they need to save their work regularly.

I added a notice reminding users to save their maps. It appears three times for each new user and also when there was no save for 10 minutes.

This solution has worked wonderfully. It's one of those features where, after shipping it, we didn't hear anything more. The support requests stopped overnight, and no one is losing their work anymore.

You can also follow me on X @hyperknot or bsky @hyperknot.com

Update, 1 Nov 2024:

This post appeared on HN, I wrote the following comment:

I'll try to collect my thoughts into one comment here about why there is no auto-save.

If you implement auto-save, users will expect every click/keystroke to be saved, like how Google Docs does it. Simply adding a 1-minute timer will make people lose work in the last minute, as they switch tabs, go offline, etc. and forget about it for days, when the browser clears tab state.

The other reason is that there is no Undo-Redo implemented on MapHub. Implementing Undo properly in a web app is not trivial if you haven't designed the whole app around a time-traveling state management. It's actually a really difficult problem to solve, even though it looks simple on the surface. Combining it with real-time collaboration is even more complex. Short story: there is no Undo-Redo.

So if you don't have Undo, then Save button plays the role of making a checkpoint, trying out something, reverting if needed. With autosave this could be ruined.

Also, you write that you expect every web app to auto-save, but this is still not the universal case today and definitely wasn't the case a few years ago. I agree that most VC-backed startups with hundred-million-dollar valuations have autosave in their web apps. Again, the proper solution is to have a time-traveling diff system implemented, which can easily get really complicated with real-time multi-user collaboration. Have a look at Figma's technical blog post about this topic.

You might be right in saying that it's not a generation of users, but the same users being conditioned to web apps doing auto-save. I agree this is probably the case. The point of the article is this definitely wasn't the case all the time, and how I've experienced this on my side, during my 8+ years of running MapHub.

I’m building a new web app and a browser extension. My previous app used server-side sessions with password login, but this time I wanted to support Single Sign-On (SSO), so I embarked on a journey to find an Auth solution.

My requirements are quite standard, or at least I thought they were:

• Never log a user out unless they want to.

• Support Google / GitHub Single Sign-On (SSO).

• Have affordable pricing for a freemium B2C app.

• As an alternative for affordable pricing, have a nice self-hosting experience.

I’ll be evaluating Auth providers from a software developer’s perspective. I’m not a security expert; I don’t have an opinion if a solution should use OAuth2 / OpenID Connect or something else. I’m looking for a setup that works reliably and gives a nice experience for my users.

Note: There is a technical distinction between authentication (AuthN) and authorization (AuthZ). Since AuthZ is super simple in my use case, I’ll be evaluating the AuthN part.

The good old setup

For MapHub, I’ve been using server-side sessions via Redis. No SSO, just password-based authentication. It’s been an extremely reliable and performant part of the infrastructure, running for almost 7 years now. Once set up, it’s been 100% stable and worry-free.

When I started writing this article, I logged into Plausible and saw that there were 19000 current visitors (Daily Mail just front-paged an article featuring a MapHub embed).

While this traffic doesn’t happen every day; I found it amazing that the single server I’ve been using for years could handle it so smoothly.

Technically, the backend is based on “boring” technologies: Pyramid (17 yr), SQLAlchemy (16 yr), PostgreSQL (26 yr) and Redis (13 yr). I believe choosing boring technology for MapHub’s backend was one of the best choices I ever made.

How about performance?

The #1 criticism about Python has always been how slow it is. The average response times during the Daily Mail Saturday evening hammer were 7-12 ms. Considering Google believes anything under 200 ms is good and 100 ms is excellent, I think these are really good numbers.

[1] curl -w “time_total” on localhost. It’s running on a dedicated AMD 3700X server.

The quest for a modern solution

7 years have passed; I am building something new and aiming to improve in these aspects:

• No more mixing of templating between Python and JS: Write everything frontend related in React and make the backend JSON API only.

• Support for Google / GitHub Single Sign-On (SSO).

• Support for running serverless functions. Just a few endpoints for long-running functions.

• And most importantly, rule #1: Do not roll my own auth. I’m looking for a stable and trusted solution made by experts in the field.

For the React + JSON API I decided to go with Next.js, but I’m trying to keep the Next.js backend part as “lite” as possible and keep using Python + SQLAlchemy for the API. (Once you’ve invested enough time to enjoy using SQLAlchemy, you don’t want to migrate to anything else.)

How did I choose which Auth solution to try out? I’ve read a lot of Hacker News and Reddit comments and tried them out in the order presented here. For some, I’ve invested 1 week+ of my time, for others, just a few hours.

Supabase

Supabase is the largest startup in this article, having received over $110 million in funding. They are making a full platform “open source Firebase alternative”, and if that’s not enough, it’s all based on PostgreSQL.

They offer amazing value: you get a managed PostgreSQL + the whole platform for $25 per month. This also includes auth for up to 100,000 monthly users, which would be hundreds or even thousands of dollars on other auth platforms.

Supabase Auth is the gateway drug to the platform. In theory, you can just use Auth, but the magic is in the interconnectedness between their systems. For example, this function inserts a new user into your tables when a user registers on Supabase Auth.

create function public.handle_new_user()
returns trigger as $$
begin
  insert into public.profiles (id, full_name, avatar_url)
  values (new.id, new.raw_user_meta_data->>'full_name', new.raw_user_meta_data->>'avatar_url');
  return new;
end;
$$ language plpgsql security definer;
create trigger on_auth_user_created
  after insert on auth.users
  for each row execute procedure public.handle_new_user();

Once you’ve started using this, you’ll never leave. It solves the number one problem with auth systems, which is how to synchronize the users of the auth system with the users of your backend system. It also creates the ultimate vendor lock-in. Luckily it’s an open-source platform, so you can always go self-hosted if you want to.

Is Supabase solid?

For Auth, unfortunately, no. I’ve been constantly getting logged out randomly from my own app, so I decided to look into how they handle the auth in detail. I quickly found out it was not an isolated case: There are many reports on GitHub about other users experiencing random logouts.

Red flag 1: No setting for session lifetime

First, there is no option to set the session lifetime. This is a very important setting on every other platform with detailed documentation. For Supabase, there is no mention of it in the docs and no setting on the UI.

This is probably the most important parameter for any auth solution and is it completely missing here?

There is a single setting called “JWT expiry limit”, which confuses users, as people are setting it to the maximum value of one week, thinking it means session lifetime. In reality, it is only setting the access token’s expiry time: a limit as short as 10 seconds should still give an OK user experience, without anyone experiencing logouts.

Red flag 2: Client-side unencrypted tokens

Every information is stored client-side. This means either non-HttpOnly cookies or localStorage. The content is not encrypted, so every script that runs on your website has access to your users’ access_token, refresh_token, provider_token and provider_refresh_token in clear text.

Now, I’m not an auth expert, so I cannot evaluate how bad this behavior is, but every other solution explicitly mentions why they are not doing this.

Red flag 3: Inconsistent, undocumented behavior, no communication

Some of the official examples (for example React) use the supabase-js package, which stores data in localStorage, while others (for example Next.js) use the auth-helpers package, which stores data in a client-side cookie. None of this is documented anywhere.

The most important information I’m trying to figure out is what happens to the auth.sessions / refresh_tokens tables on the server.

• If these are cleaned up at regular intervals, when exactly and how can we control this? This would be the missing session lifetime setting mentioned above.

• If these are not cleaned up then how do people use Supabase in production? Do they really keep all those sessions / tokens forever?

I wrote on their Discord but received no answer. I kept writing my findings in a lonely thread on their GitHub Discussions, receiving no response for 1.5 months. I opened an issue directly on the auth repo, which finally received a response after a month from a dev. The official answer is that I shouldn’t worry and that “random logouts should not be happening”. Unfortunately, they do happen, which is the reason I looked into the implementation.

Red flag 4: Lack of care for Auth

When it comes to shipping new features, Supabase is one of the most impressive companies I’ve seen. Simply reading their newsletter can be overwhelming: they ship so many features so rapidly.

At the same time, I feel there is a lack of care for one of their core packages: Auth. For example, using their Next.js middleware results in a user logout. I’m not talking about a very exotic setup, this can be reproduced on their official example. However, apart from a few users trying to guess what’s going on, no one seems to care.

Red flag 100: No 2FA on their own platform

I honestly couldn’t believe it, I just thought I couldn’t find the setting. I had to ask for confirmation about it on Supabase Discord.

There is no 2FA available for app.supabase.com!

The platform, which you are supposed to trust with your business data, your customers’ private information, and your payment processor’s API keys. The platform, which received over $110 million in funding and ships a new feature every week. They don’t have 2FA in 2023.

Note: It probably also tells a lot about Supabase Auth, which offers 2FA and is used for their platform. If they are not trusting their own solution’s 2FA, then why is it even offered?

Conclusion

I came for the Auth and wanted to stay for the platform. They offer tremendous value for $25. I really wanted to like it, I’ve spent way more time trying to make it work, compared to any other solution.

Unfortunately, there are way too many red flags for me about Supabase Auth, so I decided to look for alternatives.

I understand that their Auth platform is a fork of Netlify’s gotrue and they put a lot of effort into making it work, but the end result is still not something I’d want to lock into. Now that Supabase is a big, well-funded company with some of the best developers, they could probably develop a much better auth platform from scratch. I hope they’ll do it and become an amazing platform with a solid Auth.

Note, I’m really puzzled about how the Hacker News community universally endorses Supabase. HN is known to be very critical when it comes to discussing small to big projects, even small blog posts are scrutinized, yet they give an almost universal approval for Supabase. It was the reason I picked it first and I was the most surprised when I found the above-mentioned red flags.

NextAuth.js / Auth.js

NextAuth.js is a very popular choice these days, advertising a smooth developer experience with Next.js; however, I have still decided not to go with it.

The reasons are most evident on the GitHub Contributors graph, where I can see that:

• It’s a project that the original developer abandoned 2 years ago.

• Another developer has picked it up and is actively working on it, doing 90% of the effort today.

• The current developer’s background seems to be React and Next projects. Nothing security- or backend-related.

The other reason is that it’s not clear what they really offer. You can use a database or not. You can wrap other providers or not. You can use sessions or use JWT + JWS / JWE / JWK.

In comparison, the other solutions are full companies with a proven track record in security and they offer 1/10 of what NextAuth.js offers, developed by a single person.

I understand this is not a friendly or optimistic opinion, but choosing an Auth solution is not the same as selecting a date picker library, which you can replace in an afternoon.

I’m looking for a battle-tested auth solution developed and maintained by people who have a strong background in security. NextAuth.js didn’t pass this test for me.

Clerk

Clerk is a polished solution by a company that is 100% focused on auth. They have a solid financial background ($25 million in funding). Their website is functional and beautiful, as is their documentation.

Have a look at configuring session lifetime below: it shows exactly what’s important. A short description in the dashboard, a detailed description in the docs. Session lifetime can be set to 10 years, just what I was looking for!

They also have a super clean integration with Next.js.

Pricing and conclusion

For 10,000 active monthly users, Clerk costs $550 in the Business plan.

• If you only have paying users, and they pay you at least $5 a month, this would be max 1% of your revenue. I’d gladly pay 1% for a polished auth solution.

• On the other hand, if your business model is freemium, when only 1-2% of your users are on a paid plan, this could be 50% of your revenue. In this case, it’s clearly cost prohibitive.

In my opinion, Clerk’s Business plan is the nicest hosted auth solution if your business model fits their pricing. In my case it didn’t, so I continued my journey.

FusionAuth

FusionAuth is a company specializing in Auth, targeting bigger companies and Enterprises. They offer a self-hostable version of their software for free. It is not open-source, but you are allowed to run the binaries after accepting their license.

There is a Docker Compose version offered, so hosting it should be quite simple.

Once you start it, you get into a maze of settings. It is truly a maze. Just inside the “Edit Application” dialog, I’ve counted over 70 fields.

What I found great about FusionAuth is that they support both a login screen inside the app, as well as external login pages. Since I’m planning to develop a browser extension, I’d really prefer to have a custom-styled login screen inside the app, without pushing the users to a different subdomain first.

Having said that, I found absolutely no resources for building out this interface in React, nor for integrating their API in Next.js.

All in all, FusionAuth is probably a workable solution, but it would require investing way more time compared to alternatives. In summary, I would be spending a lot of time on the wrong priorities just to be able to use a closed-source software. I prefer to look for alternatives.

ZITADEL

ZITADEL is a Swiss company, just a drop from Munich-based Ory. Both of these companies seem to have a background in serving European government and Enterprise clients. Great news when it comes to security - probably these solutions are as battle tested as it gets.

On the other hand, it might not be such great news when it comes to setting up a simple Auth solution for a web app. Let’s see how ZITADEL scores.

Self-hosting experience

I have to say, without exception, their self-hosting experience is as smooth as it gets. First, they open-sourced their full solution, 1:1 identical to their hosted offering. This means the full backend and frontend, not just core components plus long evenings of tweaking config files.

Second, the hosting is so simple that it’s basically a single Docker container. You start a single container and you get a 1:1 identical offering to what you’d get in their Cloud offering.

There is an interesting issue on GitHub where the founder of ZITADEL and Ory are discussing their products’ differences. One key topic seems to be the self-hosting experience. In my finding, they couldn’t be further - to the advantage of ZITADEL.

The maze

Once it’s up and running, you find yourself in a smooth UI that contains hundreds if not thousands of settings. You have:

• Organizations

• Project

• Applications

• Roles

• Grants

• Authorizations

• Users

• Identity providers

• Memberships

• Actions

• Flows

• Domains

• Instances

I’ve spent a few hours with it and I couldn’t figure out how to make a simple login screen that looks like the one on their landing page:

Setting up Single Sign-On for example looks like this. Here, I had to figure out Google’s issuer is “https://accounts.google.com” and fill in every value myself:

Here is how the same screen looks in Clerk, for example:

A few clicks vs. hours of debugging how to add each of those providers, one by one.

Login screen

ZITADEL only offers sign-up/login on an external login screen, i.e. your users will be redirected to auth.company.com, where they’ll see a page hosted by ZITADEL and they’ll be redirected back to your app after logging in. The limitation of this, of course, is that the user leaves your website for an external one that has no menubar, different design, etc., and is then redirected back.

Not a huge limitation, but I prefer to have React components and implement it inside my app, especially if flows like changing password or email need to happen on the same UI.

Next.js support

ZITADEL is one of the few solutions which doesn’t provide a Next.js library. They have an example that uses NextAuth.js where there is a “provider” for ZITADEL, but then you’ve introduced a dependency both on NextAuth.js and on ZITADEL. Twice the maintenance of keeping them updated, as well as twice the attack surface.

Case in point: simply logging in with an email & password in their example app results in 5 cookies and 1 localStorage key.

I’m not looking forward to debugging random logouts on this. It was hard enough with a single cookie on Supabase.

Conclusion

The architecture of ZITADEL is probably solid, and with enough time spent, I could probably figure out all the configuration issues. Their company was responsive, they always replied immediately on GitHub in a helpful and professional way.

If they could add a native Next.js library as well as a detailed tutorial about how to get from zero to the clean login UI like they are advertising, it could even be a winner!

For now, I decided to use different solutions.

Firebase

Firebase needs no introduction, it’s been the #1 solution for “backend-as-a-service” platforms for 11 years now. It was founded in 2012, and acquired by Google in 2014 and they have been offering its services at very affordable prices ever since. Their name is so strong that Supabase’s title on their homepage is simply “open source Firebase alternative”.

Online I’ve seen most people criticizing their Functions product, but Auth seems to be a very solid solution that is polished by now.

Client-side magic

Firebase Auth seems to be the only solution I found that offers the whole auth experience client side. It really is magical: you call a single function and auth is set up and running, including UI. You don’t have to know or understand anything about Auth or backends to make it work.

Abandoned libraries

Having said that, their React lib is abandoned and is broken on React 18. Right now, the way to use it today is by browsing GitHub issues where people are trying to figure out how to make it work. One person wrote a snippet for React and another one for Next.js, which received dozens of upvotes. You copy-paste those into your code and hope it keeps working - not exactly the developer experience you’d expect from a solid platform.

Also, I cannot get over the require() inside useEffect:

useEffect(() => {
        // Firebase UI only works on the Client. So we're loading the package only after
        // the component has mounted, so that this works when doing server-side rendering.
        setFirebaseui(require('firebaseui'));  // <----- this
    }, []);

Free forever - or is it?

One of the best things about Firebase Auth is that it’s completely free for unlimited users. Probably they are the only provider which offers totally free Auth, no matter how big your project grows.

Now, if we look into the details, you can see that it’s a bit more complicated. Basically, they offer two systems, the original one “Firebase Auth” which is free, and a new one called “Firebase Auth with Identity platform”, which offers more features like MFA, Blocking functions and SAML.

Now, if you want to support a single customer needing some of this, you’d need to migrate all your users over to the new system. At which point it's going to be as expensive as other specialist auth providers.

The writing on the wall

The original Auth solution is being renamed in more and more places to “Firebase Authentication Legacy”. Now, if you open Killed By Google it doesn’t exactly give a reassuring feeling for a Google product that is already called Legacy.

I guess the legacy solution will be shut down at one point and the new “Identity platform” will become the only Firebase Auth offering. There is nothing wrong with that, but we shouldn’t think of Firebase Auth anymore as a free offering, but as a paid offering with a generous free tier (50k users!).

How do you sync user data?

Technically the biggest challenge with using Firebase Auth is how to sync your user data with your backend.

The #1 Stack Overflow answer for this question is basically “The easiest way is actually to not synchronize auth data”. Now, this might be just a single answer but catches the core idea: Firebase is really designed around the idea that you embrace Firebase as a platform, you use their DB offerings, their Functions, everything. They really don’t want you to use your own backend server with PostgreSQL and try to sync user data between the two systems.

You can find other answers which involve using Functions that trigger on Auth events, but making a reliable system based on this is not a simple task. It’d probably involve some real-time functions + a retry queue for failed events + overnight full export from Firebase with some clever diffing upon importing it.

Conclusion

Even with all the limitations, Firebase is still one of the best choices today when it comes to Auth. The abandoned React library doesn’t give a good feeling but the linked GitHub snippets work and you can get the first integration done super quickly.

Later on, it might give you headaches if you want to sync user data to your own DB, but if you are happy to use their DB it can be a solid choice.

Their free offering will probably be shut down, but their paid offering has a very generous free tier, which should cover most projects.

Supertokens

Supertokens is a relatively new startup offering hosted Auth solutions with an open-source codebase. Their self-hosting experience is really nice, there is almost nothing to configure, no web UI, just a few values. From all the solutions listed here, it might have been the simplest to get up and running.

The actual configuration happens inside your app, which seems to be a very unique solution. Here is for example the config folder of their Next.js example app.

React first

The interesting thing about Supertokens is that they especially target React developers. Their core repo has 660 commits, while their Node + React repos have almost 2000 commits together.

It has both a good and a bad side. The good side is that getting it up and running in a React codebase is literally a few clicks. They even offer their own “create-supertokens-app”, which is an interesting idea but I’d only use it to create an example app.

Implementation details all over the codebase

The bad side is that everything is super hard-wired into their way of handling React state management, which really doesn’t go well if you are using a different state management solution.

Also, there are implementation details all over your app code. For example in their official example, all the following is used in pages/:

• SuperTokensReact

• supertokensNode

• SuperTokensWrapper

• superTokensNextWrapper

• Session.attemptRefreshingSession

• Session.getSession

• verifySession

• redirectToAuth

• ThirdPartyEmailPassword.signOut

• pageProps.fromSupertokens === 'needs-refresh'

• err.type === Session.Error.TRY_REFRESH_TOKEN

• err.type === Session.Error.UNAUTHORISED

• SessionAuth

• useSessionContext

• SuperTokens.canHandleRoute

• session.getAccessTokenPayload

• supertokens-node/framework/express middleware

Also, once logged in with a simple password auth, you get 5 cookies. As I said, I’m not a security expert, but 5 cookies just feel a hint too much for me.

430 kB of JS?

The very minimalistic example implementation grows the built JS size by 435 kB! That is almost half a MB of minified JS!

I have no idea why would any auth library need almost half a MB of frontend code. And if this much code is included, why are the implementation details still scattered all over the application code?

Conclusion

I might be nitpicking here, as the setup is really simple and it works and the team was very quick to respond when I asked a question from them.

It just doesn’t give me a good feeling about how a codebase would look in the future if even the simplest possible example app looks like this. For example, in a classic, server-side rendered app (say Django or Rails), this would be 100% hidden from the developers (and also extremely well-tested by now). I understand that in a Next.js app, you have to see some of it, but this is just too much for my taste.

All in all, I decided not to use Supertokens even though I got it up and running in no time.

In the article “Why you probably do not need OAuth2 / OpenID Connect” written by the founder of Ory, he writes the following about Supertokens.

“We no longer endorse them due to a questionable choice of mixing OAuth2 terminology with self-built protocols. In particular, the use of OpenID terminology without supporting the specification itself, and the use of "access" and "refresh tokens" in incorrect contexts. We believe this to be what Scott Brady calls MyOwnOAuth™.”

Ory

Ory is an exception in this list, the reason is very simple: after trying out all these solutions, I decided to go with Ory! Having said that, the journey was far from smooth sailing.

For example, by default, it cannot be used on a public-facing website, as the client sends an HTTP request to their service even for public users. I’ve managed to fix that behavior in a Next.js proxy, but there are half a dozen similar workarounds that are still needed today.

In the end, I decided to stay with Ory because their team was very professional and I believe those issues will be addressed soon. In my opinion, the core service is very solid, just the integration parts are having some rough edges.

I’ve decided to write my experience with Ory in a dedicated blog post, which I’ll be posting next. If you don’t want to miss it, you can subscribe here.

Subscribe now

The ones I haven’t tried: Auth0, Keycloak, WorkOS

Auth0 is the “Nobody ever got fired for buying IBM” product of Auth solutions. It’s unreasonably expensive, yet is the standard solution for Enterprise companies. Their pricing is not public above 10k MAU, but from what I’ve read they are the only company that raises the per-user cost instead of lowering it as you get bigger. Instead of offering volume discounts, you get volume “punishment”.

They are at least super safe, right? No, they got hacked twice in the last 12 months.

KeyCloak is the open-source alternative, but even seasoned ops professionals are talking about configuring it like wartime stories.

WorkOS - if I understand correctly - is only providing Enterprise features, like SAML. That is, it’d be used in addition to your regular auth solution, thus is not what I was looking for. BTW, they have a really strong frontend team, they are behind Radix UI.

Synchronization

Before I conclude this post, I want to write about the issue with synchronization: Where do you store your user’s login details?

In a classic, server-side framework like Django or Rails, of course, everything is in the production DB. You have foreign keys and everything is guaranteed to be correct and up-to-date.

Now, what happens when you start using a hosted auth solution? Half of your user’s data (email, password, SSO tokens) has to live in their DB, while the other half (the actual user-created data, subscription info, etc.) lives in your DB.

As I’ve written for Firebase, I believe constantly synchronizing these two DBs is not an easy feat. Most hosted auth products have a JSON API and webhooks that allow you to build such a solution, but it’s one of those areas where I can imagine even years into a startup’s life you will encounter bugs and edge cases.

Now, with a self-hosted version, you can actually share the same DB, as all self-hosted solutions support PostgreSQL. That alone is an amazing selling point in my opinion. Thus strangely, a paid option might be more complicated in the long run compared to a self-hosted one.

This is one aspect where Supabase is above all the other hosted solutions: their auth uses the same DB as your application server, hence it’s a hosted auth solution where the DB is shared with your app DB.

Conclusion

I’ve spent way more time finding an auth solution than I originally wanted, but I think I’ve learned a lot in the process. I started with Supabase but even after weeks of trying to make it work, the experience was so bad with random logouts that I just couldn’t consciously choose it.

After reviewing many options, I recommend the following:

• If your project can afford the pricing of Clerk, go with them.

• If not and you have backend / devops experience, choose Ory. I’ll write a guide for it in the next post.

• Otherwise, choose Firebase. Be aware though that the free version will probably be deprecated in the future.

If you are interested in my experience with Ory, feel free to subscribe here, it’ll be my next post.

You can also contact me on X, bsky or in email.

Update 12th Nov 2024: There was a great HN discussion about this article.