>_C2CZ
Serverless · Self-Hosting · Security · Independent Ops

dh key too small: OpenSSL 3 rejects 1024-bit DH params

Kai Tanaka covers SSH, TLS and applied cryptography at C2CZ. Author page · All authors.

curl: (35) TLS connect error: error:0A00018A:SSL routines::dh key too small is OpenSSL 3 refusing a handshake, not a network failure, and it is aimed at the server’s ephemeral Diffie-Hellman parameters rather than at your certificate store. OpenSSL 3 ships with security level 2 as the library default, which rejects any DH key shorter than 2048 bits, so every client linked against it — curl, Python, Node, PHP, the PHP that powers half the appliances you are scanning — drops the connection before a single byte of HTTP is exchanged. The device on the other end is usually a switch, a camera, an IP-KVM, a BMC or a five-year-old NAS that will never be updated; you are the one who has to move.

Everything quoted below was executed on one arm64 Linux 6.17 host with OpenSSL 3.5.6, curl 8.14.1, Python 3.13.5 and Node 26.5.1 (its own OpenSSL 3.5.7). The lab is a local openssl s_server holding 1024-bit DH parameters, which reproduces both halves of the failure: the peer that offers the small key, and the client that refuses it. Error codes, the transcript lines and the working cipher string are pasted from those runs.

TL;DR — the 1024-bit refusal in one screen

  • The default security level is 2. OpenSSL 3 rejects RSA, DSA and DH keys shorter than 2048 bits at that level. The parameter at fault here is dhparam (the ephemeral DH group), not the certificate.
  • Two error codes, one cause. Client side, when the peer’s ServerKeyExchange carries a small group: 0A00018A … tls_process_ske_dhe:dh key too small. Server side, when your own config tries to load such a group: 0A00018A … SSL_CTX_set0_tmp_dh_pkey:dh key too small. A 1024-bit certificate fails differently again: 0A00018F … SSL_CTX_use_certificate:ee key too small.
  • It only bites on TLS 1.2 and below. TLS 1.3 dropped finite-field DHE from the handshake, so a DHE-1024 peer over TLS 1.3 fails on protocol version instead, if it fails at all.
  • The client-side escape hatch is one token. Append @SECLEVEL=1 to the cipher string — DEFAULT@SECLEVEL=1, or <cipher>@SECLEVEL=1 — and the same handshake completes. Measured on all four clients below.
  • Know what else level 1 unlocks before you ship it. Level 1 permits 1024-bit RSA/DSA/DH and SHA-1 signatures. Level 0 permits MD5 and 512-bit keys. Scope the downgrade to the one connection if you possibly can.
  • The durable fix is on the peer — regenerate its DH group at 2048 bits or move the service to ECDHE. Read the peer’s key size first; s_client -brief prints Peer Temp Key: DH, 1024 bits and settles the question in one line.

What OpenSSL 3 is actually checking

The security level is a single integer on the SSL_CTX, and it is consulted for every algorithm, key, parameter and protocol version the connection wants to use. It is not a cipher-list preference. Setting CipherString = ALL does not override it: the documentation states that the limit applies no matter what other settings the application has made, so only suites consistent with the level are permitted.

This is why the same failure shows up in a dozen different messages. The level polices cipher suites, ECC curves, signature algorithms, DH parameter sizes and certificate key sizes simultaneously. The relevant rows of the default callback, as documented for OpenSSL 3.5:

LevelSecurityRejectsAlso
0—nothingOpenSSL 1.x compatibility behaviour
180 bitsRSA / DSA / DH < 1024 bits, ECC < 160 bitsMD5 MACs banned, SHA-1 and MD5 signatures banned, TLS 1.0/1.1 and SSLv3 disabled
2 (default)112 bitsRSA / DSA / DH < 2048 bits, ECC < 224 bits+ RC4 banned, compression disabled
3128 bitsRSA / DSA / DH < 3072 bits, ECC < 256 bits+ no forward secrecy, session tickets off

Two consequences worth internalising before the first fix attempt. First, level 1 is not “everything old”: TLS 1.0 and TLS 1.1 stay disabled, which is why a legacy peer that can only speak TLS 1.0 will still fail after you drop to level 1. Second, the level applies to both ends of the handshake. A client at level 2 hitting a server at level 1 produces the client-side error; a server at level 2 loading small parameters fails at startup, before any client appears.

Reproduce both halves in under a minute

You need three files and no network: a throwaway certificate, a small DH group, and a server willing to offer it. Generating a 1024-bit group is fast; a 2048-bit group takes a few seconds and is what you will end up shipping anyway.

mkdir -p /tmp/dh-lab && cd /tmp/dh-lab
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 30 -nodes \
  -subj "/CN=legacy-lab.local"
openssl dhparam -out dh1024.pem 1024

Now try to serve it with the library defaults, exactly as an untouched install would. It does not start. The refusal names the parameter, the function, and the security check:

openssl s_server -accept 8443 -cert cert.pem -key key.pem -dhparam dh1024.pem \
  -tls1_2 -cipher 'DHE-RSA-AES256-GCM-SHA384' -quiet
Error setting temp DH parameters
20A01EDC75E20000:error:0A00018A:SSL routines:SSL_CTX_set0_tmp_dh_pkey:dh key too small:../ssl/ssl_lib.c:7560:

That is the server-side half, and it is the sentence to look for in nginx or stunnel startup logs after an OpenSSL 3 upgrade: the daemon refuses to boot, and the parameter file is the culprit. To continue the experiment you have to force the server down to level 1, which makes the security level part of the cipher string:

openssl s_server -accept 8446 -cert cert.pem -key key.pem -dhparam dh1024.pem \
  -tls1_2 -cipher 'DHE-RSA-AES256-GCM-SHA384@SECLEVEL=1' -www

With that peer up, every default-configured client on the box refuses it. Four separate tools, four different wrappers around the same OpenSSL check:

Client (version)Default — security level 2With @SECLEVEL=1
curl 8.14.1 / OpenSSL 3.5.6curl: (35) TLS connect error: error:0A00018A:SSL routines::dh key too smallhandshake completes: TLSv1.2 / DHE-RSA-AES256-GCM-SHA384
openssl s_client 3.5.6error:0A00018A:SSL routines:tls_process_ske_dhe:dh key too small:../ssl/statem/statem_clnt.c:2354:CONNECTION ESTABLISHED, Peer Temp Key: DH, 1024 bits
Python 3.13.5 / OpenSSL 3.5.6ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1029)OK TLSv1.2 DHE-RSA-AES256-GCM-SHA384
Node 26.5.1 / OpenSSL 3.5.7EPROTO: write EPROTO …tls_process_ske_dhe:dh key too small:../deps/openssl/openssl/ssl/statem/statem_clnt.c:2354:HTTP 200 TLSv1.2 DHE-RSA-AES256-GCM-SHA384

Note the stack frames in the middle column. Node reports ../deps/openssl/openssl/… because it links its own OpenSSL build, while curl and Python report the system path — the same check, compiled in twice. That difference is also why two tools on one machine can disagree: a brew or conda Python with a newer OpenSSL fails where the system Python 3.11 with LibreSSL or OpenSSL 1.1 succeeds.

The client-side fix, per tool

If you only need to finish the job — a firmware upload, a config read, a one-off pull from a device that will be scrapped — the switch is per-connection and does not touch the library default.

With curl, the modifier rides on --ciphers, and it must be paired with a TLS 1.2 cap or a DHE cipher that the level-2 handshake would have negotiated anyway. --ciphers affects TLS 1.2 and below only; TLS 1.3 suites live in --tls13-ciphers, which is why a level-1 workaround and a TLS 1.3-only connection are mutually exclusive ideas:

curl -v --tls-max 1.2 --ciphers 'DEFAULT@SECLEVEL=1' -k https://127.0.0.1:8446/

With the openssl command itself, the modifier can be appended to a single cipher, which is the narrowest form available — everything else in the connection stays at level 2:

openssl s_client -connect 127.0.0.1:8446 -tls1_2 \
  -cipher 'DHE-RSA-AES256-GCM-SHA384@SECLEVEL=1' -brief

In Python, set_ciphers() takes the same string, and the failure surfaces as an ssl.SSLError whose reason is DH_KEY_TOO_SMALL — catch it explicitly rather than pretending an OSError handler covers TLS policy. This probe is the one that produced the table above:

import socket, ssl, sys

HOST, PORT = "127.0.0.1", int(sys.argv[1]) if len(sys.argv) > 1 else 8446

def probe(cipher_string, label):
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
    ctx.check_hostname = False
    ctx.verify_mode = ssl.CERT_NONE          # lab self-signed cert only
    ctx.minimum_version = ssl.TLSVersion.TLSv1_2
    ctx.maximum_version = ssl.TLSVersion.TLSv1_2
    try:
        ctx.set_ciphers(cipher_string)
    except ssl.SSLError as exc:
        print(f"[{label}] set_ciphers failed: {exc}")
        return
    try:
        with socket.create_connection((HOST, PORT), timeout=5) as raw:
            with ctx.wrap_socket(raw, server_hostname="legacy-lab.local") as tls:
                print(f"[{label}] OK {tls.version()} {tls.cipher()[0]}")
    except ssl.SSLError as exc:
        print(f"[{label}] SSLError: {exc}")
    except OSError as exc:
        print(f"[{label}] OSError: {exc}")

print(f"openssl: {ssl.OPENSSL_VERSION}")
probe("DHE-RSA-AES256-GCM-SHA384", "seclevel=2 (library default)")
probe("DHE-RSA-AES256-GCM-SHA384@SECLEVEL=1", "seclevel=1 (opt-in)")

Node takes the string as an agent option, and the error arrives as EPROTO on the request object, not as a certificate error. Read the negotiated parameters inside the response callback: this lab’s server closes the socket before the response end event fires, and getCipher() returns null on a dead socket — a trap that turns a working connection into a TypeError in monitoring code.

import https from "node:https";

const port = Number(process.argv[2] ?? 8446);

function probe(ciphers, label) {
  return new Promise((resolve) => {
    const req = https.request(
      {
        host: "127.0.0.1", port, path: "/", method: "GET",
        rejectUnauthorized: false,          // lab self-signed cert only
        minVersion: "TLSv1.2", maxVersion: "TLSv1.2",
        ciphers,
      },
      (res) => {
        const proto = req.socket.getProtocol() ?? "?";
        const cipher = req.socket.getCipher()?.name ?? "?";
        res.resume();
        res.on("end", () => resolve(`[${label}] HTTP ${res.statusCode} ${proto} ${cipher}`));
      },
    );
    req.on("error", (err) => resolve(`[${label}] ${err.code ?? "ERR"}: ${err.message}`));
    req.end();
  });
}

console.log(`node ${process.version} / ${process.versions.openssl}`);
console.log(await probe("DHE-RSA-AES256-GCM-SHA384", "seclevel=2 (library default)"));
console.log(await probe("DHE-RSA-AES256-GCM-SHA384@SECLEVEL=1", "seclevel=1 (opt-in)"));

Runtimes that do not use OpenSSL behave differently, and that asymmetry is useful during triage: a Go or Rust client using crypto/tls or rustls implements its own key-exchange checks and has no @SECLEVEL to lower, so “curl fails, our Go service works” is a coherent report rather than a broken test rig.

When the server is the one refusing

Self-hosters meet the same error code from the other seat. After an Ubuntu or Debian release upgrade pulls in OpenSSL 3, an nginx, apache2, haproxy, stunnel or dovecot instance that has been serving a 1024-bit dhparam.pem for years reloads and dies. The log line is the one measured above: SSL_CTX_set0_tmp_dh_pkey:dh key too small. Nothing is wrong with the certificate, the key or the config syntax — the parameter file is simply below the floor.

A second, easily confused variant belongs to the certificate itself. Swap in a 1024-bit RSA key and the server refuses at startup with a different error code — the same family, a different asset:

openssl req -x509 -newkey rsa:1024 -keyout key1024.pem -out cert1024.pem -days 30 -nodes \
  -subj "/CN=rsa1024-lab.local"
openssl s_server -accept 8445 -cert cert1024.pem -key key1024.pem -tls1_2 -www
error setting certificate
2070FA346EFD0000:error:0A00018F:SSL routines:SSL_CTX_use_certificate:ee key too small:../ssl/ssl_rsa.c:252:

Triage rule: 0A00018A or the words dh key mean the ephemeral group — regenerate dhparam. 0A00018F or ee key mean the leaf certificate — reissue it. Both are level-2 refusals rather than corruption, and both are silent about their cause if the application does not check return values; the documentation warns that an ignored error makes it look as though no certificate was set at all.

Read the peer’s DH size before you loosen anything

The one measurement that decides your course is the size of the group the peer actually offers. s_client -brief prints it, and it costs a single connection:

openssl s_client -connect 127.0.0.1:8446 -tls1_2 -cipher 'DHE-RSA-AES256-GCM-SHA384@SECLEVEL=1' -brief
Protocol version: TLSv1.2
Ciphersuite: DHE-RSA-AES256-GCM-SHA384
Peer Temp Key: DH, 1024 bits

Three readings matter. Peer Temp Key: DH, 1024 bits at level 1 but a dh key too small failure at level 2 confirms the group is the whole problem and a 2048-bit regeneration upstream would end it. A handshake that succeeds at level 2 with no Peer Temp Key line at all means the peer chose ECDHE or RSA key exchange and the error you are chasing comes from elsewhere. A peer that fails at level 1 with the same message is offering something below 1024 bits, which puts you at a different decision: level 0, or a compensating control, or replace the device.

The durable fix, and the one you should scope

On a peer you control, regenerating the group is a two-minute change and better than any client-side workaround. 2048 bits satisfies level 2 and stays inside every modern client’s expectations; the file is the only thing that changes.

openssl dhparam -out /etc/nginx/dhparam.pem 2048
openssl dhparam -in /etc/nginx/dhparam.pem -text -noout | head -1

For nginx the file is wired in with ssl_dhparam (and proxy_ssl_dhparam when nginx is the client side of a proxied TLS hop); the directive names are the ones in nginx’s own documentation, and the value you point at is what changes. Skip dhparam entirely if you can move to ECDHE: an ECDHE-only cipher list never consults the DH file, and the class of error disappears rather than being tuned.

The tempting shortcut is a machine-wide downgrade in /etc/ssl/openssl.cnf, where the system default security level can be pinned for every OpenSSL-linked binary at once. Understand what you are buying: level 1 also un-bans SHA-1 signatures and permits 1024-bit RSA certificates, so this is a fleet-wide weakening to unblock one appliance — to be applied, if ever, in a lab or an isolated management VLAN, never on a host that terminates public TLS.

Where the client-side escape hatch is legitimate, keep it local: on the command line for a one-off, or in the one code path that talks to the legacy device, with a comment naming the device and the date you expect it to disappear. A DEFAULT@SECLEVEL=1 in a shared HTTP client configuration silently downgrades every other endpoint that client reaches, including the ones that never needed it.

If your OpenSSL 3 migration is producing other refusals, the two close relatives are worth reading next: the legacy-provider error that breaks MD4 and old ciphers, and the certificate-chain failure that curl reports as error 60 — a different error class that people misread as the same “OpenSSL got stricter” problem. If the peer is a device you reach over a tunnel rather than the open internet, the mTLS alert post covers the handshake alert that arrives when a client certificate is required and missing.

Verification checklist

  • Name the asset. dh key in the message → ephemeral DH group. ee key → certificate. Neither → relaunch with -brief or check the peer’s suite.
  • Confirm the level. Level 2 is the OpenSSL 3 default and rejects < 2048-bit DH/RSA/DSA. If a machine behaves like level 1, something in openssl.cnf or the environment has pinned it there.
  • Check the protocol. A TLS 1.3 connection ignores DHE parameters entirely; force --tls-max 1.2 before concluding anything about DH.
  • Scope the downgrade. @SECLEVEL=1 belongs on one cipher string or one code path, dated and commented.
  • Re-test from a second tool. If curl and Node disagree, compare their OpenSSL versions first — two linked copies are two independent policies.
  • Measured on this host: arm64 Linux 6.17.0, OpenSSL 3.5.6, curl 8.14.1, Python 3.13.5, Node 26.5.1 with OpenSSL 3.5.7. Every terminal transcript above is that lab’s stdout, including both server-side startup refusals.
  • OpenSSL, SSL_CTX_set_security_level — the default callback table (level 1 = 80 bits, level 2 = 112 bits and “RSA, DSA and DH keys shorter than 2048 bits … are prohibited”, level 3 = 3072 bits) and the note that the limit applies regardless of custom cipher settings: docs.openssl.org/3.5/man3/SSL_CTX_set_security_level/.
  • nginx, ssl_dhparam and proxy_ssl_dhparam directive documentation (config referenced above was not executed on this host): nginx.org/en/docs/http/ngx_http_ssl_module.html.

C2CZ

Hyper-specific engineering guides: Cloudflare Workers, Turso, self-hosting, network security, offensive security, independent operations.