What could cause "dh key too small" error?

The best explanation I found is on Debian wiki at https://wiki.debian.org/ContinuousIntegration/TriagingTips/openssl-1.1.1 which says:

In Debian the defaults are set to more secure values by default. This is done in the /etc/ssl/openssl.cnf config file. At the end of the file there is:

[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

This can results in errors such as:

dh key too small
ee key too small
ca md too weak

This is caused by the SECLEVEL 2 setting the security level to 112 bit. This means that RSA and DHE keys need to be at least 2048 bit long. SHA-1 is no longer supported for signatures in certificates and you need at least SHA-256. Note that CAs have stopped issuing certificates that didn't meet those requirements in January 2015, and since January 2017 all valid CA certificates should meet those requirements. However there are certificates generated by private CAs or that are in a test suite that do not meet those requirements.

SECLEVEL 1 was the default in previous versions and is at the 80 bit security level, requiring a 1024 bit RSA key.

The problem of "dh key too simple" is fully detailed at https://weakdh.org/ It says:

Diffie-Hellman key exchange is a popular cryptographic algorithm that allows Internet protocols to agree on a shared key and negotiate a secure connection. It is fundamental to many protocols including HTTPS, SSH, IPsec, SMTPS, and protocols that rely on TLS.

We have uncovered several weaknesses in how Diffie-Hellman key exchange has been deployed:

[..]

What Should I Do? If you run a server…

If you have a web or mail server, you should disable support for export cipher suites and use a 2048-bit Diffie-Hellman group.

What to do?

Try not to change SECLEVEL. It is better to keep it systemwide as it is (SECLEVEL=2), but try instead to fix the specific connection problem you have.

First step is to contact the administrator of the service you are trying to connect to over TLS, and give it the details above so that he does the necessary change to not be anymore in the "weak dh" vulnerability.

If that proves impossible, or in the mean time before resolution, just set, for the connection, the "SSL" ciphers to the value of DEFAULT@SECLEVEL=1. This unfortunately lowers the security but at least it impacts only this specific connection and not your whole system. You do not need to hardcode the list of ciphers the value above will make things work exactly as before.

I had that exact same problem and resolved it that way, because it couldn't convince the remote party to change its TLS server or provide certificates at least using RSA 2048 bits.

This is an issue with the program if you have not disallowed weak and vulnerable ciphers.

DO NOT reduce the seclevel.

A small DH key can expose the TLS communication to vulnerabilities such as LogJam.

You would need to specify a strong cipher set in your program. For a quicj test try "HIGH:!DH:!aNULL"

You could also specify a strong set of cipher suites such as in:

"ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"

But for this to happen, the server needs to be hardened to support these suites as well. I'm assuming that is already the case and your local openssl is the latest as well and supports these.

Courtesy on cipher suites list: https://www.acunetix.com/blog/articles/tls-ssl-cipher-hardening/