>_C2CZ
Serverless · Self-Hosting · Security · Independent Ops

Tailscale SSL_ERROR_RX_RECORD_TOO_LONG: Fix HTTPS

Firefox throws SSL_ERROR_RX_RECORD_TOO_LONG on a https://machine.tailnet.ts.net URL when something on the machine's port 443 answers with plain HTTP instead of TLS. I hit this twice in the same week — once on a NAS after running tailscale cert, once after uninstalling a reverse proxy that had been squatting on 443 — and both times the fix was about who owns the port, not about the certificate files themselves.

This is a Firefox-specific rendering of a generic failure: a TLS client sent a ClientHello and got back bytes that are not TLS. Chrome shows ERR_SSL_PROTOCOL_ERROR for the same condition, and curl reports wrong version number — I reproduced all three against a local plain-HTTP server on port 443 before writing this. Here is the diagnostic order that finds the culprit, and the exact fixes for the three Tailscale setups that produce this error.

TL;DR

  • SSL_ERROR_RX_RECORD_TOO_LONG = the port answered, but not with TLS. The server on 443 spoke plain HTTP to a TLS ClientHello. A closed port gives "connection refused", not this error.
  • On Tailscale, the cause is one of three things: HTTPS Certificates not enabled for the tailnet, tailscale cert output never installed into the actual service, or another process (leftover reverse proxy, container) squatting on port 443.
  • Diagnose in two commands: curl http://host:443/ (200 = plain HTTP on 443, 400 = TLS port correctly rejecting HTTP) and sudo ss -tlnp | grep :443 to see who owns the port.
  • Fix the root cause — enable HTTPS in the admin console, point tailscale serve at the app, or remove the squatter. Do not just click past the warning.

What SSL_ERROR_RX_RECORD_TOO_LONG actually means

The error comes from NSS, Firefox's TLS library. Its official text is "SSL received a record that exceeded the maximum permissible length." Every TLS message starts with a five-byte record header: one byte for content type, two bytes for the protocol version, two bytes for the payload length. Firefox reads those five bytes and waits for exactly that many more.

When the server answers with a plain HTTP response, the first five bytes are HTTP/ — 0x48 0x54 0x54 0x50 0x2F. Firefox parses bytes 3–4 (0x50 0x2F) as a record length of 20,527 bytes. TLS allows at most 18,432 bytes per record (TLS 1.2) and 16,640 (TLS 1.3), so the connection is dropped before certificates, caches, or clocks ever enter the picture. It is a server-side misconfiguration, and it is decided in the first five bytes of the reply.

That is why this error is almost never about the certificate content. An expired certificate produces a date error; an untrusted one produces SEC_ERROR_UNKNOWN_ISSUER. SSL_ERROR_RX_RECORD_TOO_LONG means the handshake never got far enough to look at a certificate — the reply was not TLS at all.

Why Tailscale produces it

MagicDNS gives every node a clean machine.tailnet.ts.net hostname, but the hostname is only half of HTTPS. The other half is something terminating TLS on port 443, and there are exactly three setups where that something is missing or wrong:

SetupWhat is on port 443Result
HTTPS Certificates disabled on the tailnetYour app's plain HTTP (or nothing with TLS configured)ClientHello gets an HTTP reply
tailscale cert ran, certs never installedYour app still serving plain HTTP on 443ClientHello gets an HTTP reply
A leftover reverse proxy / container from beforenginx, Caddy, or Traefik answering HTTP on 443ClientHello gets an HTTP reply (often a redirect)

The third one is the sneakiest. I had removed a Caddy container weeks earlier and still got the error — the container was running in the background, answering 443 with an HTTP 308 redirect to HTTPS, which Firefox read as garbage TLS. The same story shows up in the tailscale/tailscale issue tracker (#13276): HTTPS broke after "uninstalling" Caddy until the process was actually killed.

Diagnose it in two commands

First, confirm the port is serving plain HTTP. Send an unencrypted request to port 443 and read the result the opposite way round from normal:

curl -sS -o /dev/null -w '%{http_code}\n' http://machine.tailnet.ts.net:443/

A 200 is the bad outcome — the port answered a plain-HTTP request with real content, which is exactly what Firefox received. A 400 is the healthy outcome: that is what nginx and Apache return when plain HTTP arrives on a TLS port ("The plain HTTP request was sent to HTTPS port" in nginx, "You're speaking plain HTTP to an SSL-enabled server port" in Apache). If you see 400, your port is terminating TLS correctly and this error is coming from somewhere else — check the URL scheme and any local proxy or antivirus that intercepts connections.

Second, find who owns the port:

sudo ss -tlnp | grep :443

An empty result means nothing is listening — you would get "connection refused", not this error, so that is not your case. The process shown is the one answering Firefox. If it is not the service you configured (a NAS web UI, tailscaled, your app), that process is the problem. When the answer is your app, the fix is below; when it is nginx or a container you thought you removed, see the next section.

Fix 1: Enable HTTPS Certificates and let tailscale serve terminate TLS

If you want https://machine.tailnet.ts.net to work with zero manual certificate management, enable the HTTPS Certificates feature and proxy through tailscale serve. This is the least-friction fix and the one I recommend for self-hosted apps.

In the admin console: open DNS → enable MagicDNS if it is off → under HTTPS Certificates, select Enable HTTPS. Tailscale will warn that your machine names and tailnet DNS name are published on a public certificate ledger — that is how the ACME-style issuance works, and it is the documented trade-off (Tailscale docs, "Enabling HTTPS"). Then serve your local app:

sudo tailscale serve --bg --https=443 http://127.0.0.1:8080
sudo tailscale serve status

--bg keeps the proxy running in the background; --https=443 binds the HTTPS listener; the target is your app's local HTTP port. Tailscale obtains and rotates the certificate for your MagicDNS name automatically. Access the app at https://machine.tailnet.ts.net and the error is gone.

One wrinkle: if your backend itself speaks HTTPS with a self-signed certificate (common with NAS web UIs and Dockerized apps that terminate their own TLS), point serve at the https+insecure:// scheme instead — this is the officially accepted target form for untrusted local backends, confirmed in tailscale/tailscale #18849:

sudo tailscale serve --bg --https=443 https+insecure://127.0.0.1:8443

Prefer http:// for loopback traffic when the app allows it — you avoid double encryption on a link that never leaves the machine. Reach for https+insecure:// only when the app refuses plain HTTP. Traffic between your device and the node is still encrypted by WireGuard; the "insecure" applies only to the localhost hop.

Fix 2: Install the certificate into the service that owns port 443

If you would rather terminate TLS inside your app (NAS web UI, Synology DSM, custom service), tailscale cert alone is not enough. Running it only downloads the certificate and key into the current directory:

sudo tailscale cert machine.tailnet.ts.net
# writes machine.tailnet.ts.net.crt and machine.tailnet.ts.net.key
# into the current directory

The command does not know what you planned to do with those files — it does not install them anywhere. This is the exact trap from the Stack Overflow question this post is named after: a NAS owner enabled MagicDNS and HTTPS, ran tailscale cert, and still got SSL_ERROR_RX_RECORD_TOO_LONG because the NAS web server kept serving plain HTTP on 443 with its default certificate. You must copy the files into the service and make it use them — for a Synology NAS that means Control Panel → Security → Certificate → add the .crt/.key pair and set it default; for an nginx-terminated app, wire them into the server block:

server {
    listen 443 ssl;
    server_name machine.tailnet.ts.net;

    ssl_certificate     /etc/ssl/ts/machine.tailnet.ts.net.crt;
    ssl_certificate_key /etc/ssl/ts/machine.tailnet.ts.net.key;

    location / {
        proxy_pass http://127.0.0.1:8080;
    }
}

Then reload the service and re-test with the curl http://…:443/ diagnostic — it must return 400, proving the port now speaks TLS. If you handle TLS yourself, remember the certificate has a limited lifetime and Tailscale rotates it; plan a renewal mechanism (a cron job re-running tailscale cert and reloading the service is the standard pattern).

Fix 3: Remove the process squatting on port 443

The reverse-proxy case. Your ss -tlnp output shows nginx, Caddy, or Traefik listening on 443 — usually left over from an old setup, or a container that restarted with the daemon. Stop it, disable it, and make sure the container is really gone:

sudo systemctl stop caddy nginx
sudo systemctl disable caddy nginx
sudo docker ps --filter "publish=443"   # find containers exposing 443
sudo docker rm -f <container-name>

Verify the port is free (or owned by the right process) before testing again:

sudo ss -tlnp | grep :443
# empty = free; or shows only the service you intend

Then re-run sudo tailscale serve --bg --https=443 http://127.0.0.1:8080 or your app's own listener. Do not skip the port check and restart tailscaled first — if the squatter still holds 443, serve will fail to bind and you will be debugging a different error. If you are rebuilding an nginx frontend instead of removing it, our earlier write-up on nginx 499 client closed request covers the timeout chain you will want configured correctly; and for auditing every listener on the box before you trust the port map, the Nmap root-privileges fix has the exact flags for a non-root SYN scan.

Verify the fix

Three checks, in order, and the error is genuinely dead:

  1. Port speaks TLS: curl -sS -o /dev/null -w '%{http_code}\n' http://machine.tailnet.ts.net:443/ returns 400 (plain HTTP rejected).
  2. HTTPS answers: curl -sSI https://machine.tailnet.ts.net/ returns 200 (or your app's expected code) with no TLS errors.
  3. Certificate is Tailscale's: echo | openssl s_client -connect machine.tailnet.ts.net:443 -servername machine.tailnet.ts.net 2>/dev/null | openssl x509 -noout -subject -issuer -dates shows the ts.net subject and a valid date range.

Then open the URL in Firefox. If you still see the error after all three checks pass, the interception is client-side — a corporate proxy, antivirus TLS scanner, or browser extension answering the request with its own plain-HTTP page. Test from a phone on the tailnet (Tailscale app) to rule the client out.

One last note: don't be tempted to add a security exception and move on. This error means a port that should be encrypted is not — on a Tailscale node that is a red flag worth ten minutes of diagnosis, not a warning to click through.

C2CZ

Hyper-specific engineering guides: Cloudflare Workers, Turso, self-hosting, network security, offensive security, independent operations.