The bug that only happened in production

Here's what turned out to be wrong, which is more interesting than it sounds, and what we did about it.

Four combinations, twenty minutes

The tempting thing with a bug like this is to start guessing. Local network permissions. Apple's rules about insecure connections. CORS, the mechanism that lets a server say which websites are allowed to call it. All three are plausible, and each one would have taken an afternoon to chase down.

So we tried every combination instead. Two browser engines, two addresses the page could be loaded from:

One failing cell out of four, and the other three ruled out most of the suspects on their own. The same WebKit view happily makes the same insecure request to the same port when the page is served over plain http, so it isn't a network permission and it isn't Apple's transport rules - those don't care which address the page came from. Chrome reaches the server from the production address, so the local server is perfectly happy to be called by our page, which rules out CORS.

Which leaves one explanation: WebKit, on an https page, refusing to make a request to http://localhost.

Why WebKit says no

Browsers block an https page from loading things over plain http. That rule is correct and nobody's asking for it back.

The specification carves out an exception for addresses that point at your own machine, on the grounds that a request which never touches a network can't be intercepted on the way. Chromium implements the exception. Firefox implements it. WebKit never has, and the bug has been open since 2017.

Our production app loads https://snownotes.co and renders it in a WebKit view, so every request to http://localhost:1234 was being stopped before it left the page.

The part that cost us the most time is worth saying plainly, because it's exactly the wrong thing to assume when you're debugging: our Content Security Policy already allowed the request. That policy is the list of places a page is permitted to fetch from, and ours named http://localhost:* explicitly. It made no difference, because permitting something in that list and the mixed-content check are two separate gates, and the second one still gets a vote. Adding a permission there doesn't grant anything; it only removes a restriction we'd otherwise have imposed ourselves.

There was a smaller problem underneath. The code that checks whether a provider is reachable caught the exception and reported the provider as unreachable, so a policy decision made by the browser arrived looking like a server that wasn't running, with nothing written down anywhere that anyone could read. We'd probably have found the real cause in an hour if the error had said what it was.

The fix: leave the webview

If the browser engine won't make the request, something else has to. Our native apps are built with Tauri, which means there's a Rust process sitting underneath the web page, and that process has no opinion about mixed content. So the request goes there instead. The page hands it over, Rust makes the connection, and the response streams back in.

The decision worth drawing attention to isn't the transport, though. It's that every native app now uses it, including the ones that were never broken.

Leaving the two working platforms on their own quiet path would have saved a day that week and cost us a little every week after.

Consent, not a backdoor

Letting a web page make arbitrary network requests through native code is a bad idea if you're careless about it. A compromised page shouldn't get a readable window into somebody's home network.

So the native path isn't a general-purpose HTTP client. Requests to your own machine go through without ceremony, since that's the ordinary case and nothing leaves the computer. Anything else, like a model running on another box in your flat, has to be approved by you once, and we remember the answer. Redirects aren't followed, so a host you approved can't bounce the request somewhere you didn't.

Where this works today

The honest version, since this is the part such a post usually skips.

What we took from it

Two things, one specific and one less so.

The specific one: when a platform behaves differently from a specification you've read, check whether it implements the part you're leaning on. We had a comment in our config asserting that requests to your own machine are exempt from mixed-content blocking. It was true of the engines whoever wrote it had tried, and false of the one that broke.

The general one: run the whole matrix before you form a theory. Those four combinations took about twenty minutes and eliminated three explanations that would each have cost an afternoon. Starting from a single failing case is how you end up carefully fixing something that was never broken.

We're in closed beta while we work through more of this. If you'd like an invite, join the waitlist.