How to fix "TrustAnchor is not a CA certificate" for self-signed certificates? (Java)

user_14102009 0
Viewed 1749

I'm trying to create a self-signed server certificate for my client/server application (written in Java using the Netty library). From my understanding, I have to create a CA keystore and a server keystore, sign the server certificate using the CA keystore and place the CA certificate in the client's truststore. For this purpose, I have written the following windows script:

call keytool -genkey -alias ca -keyalg RSA -keystore ca.keystore -storetype JKS -storepass changeit -keypass changeit
:: create CA keystore

call keytool -exportcert -rfc -alias ca -file truststore.pem -keystore ca.keystore -storepass changeit
:: export CA certificate

call keytool -genkey -alias server -keyalg RSA -keystore server.keystore -storetype JKS -storepass changeit -keypass changeit
:: create server keystore

call keytool -certreq -alias server -keystore server.keystore -file server_signing_request.csr -storepass changeit
:: ask CA to sign server certificate

call keytool -gencert -infile server_signing_request.csr -outfile signed_server_cert.crt -keystore ca.keystore -alias ca -storepass changeit
:: sign server certificate

call keytool -importcert -alias ca -keystore server.keystore -file truststore.pem -storepass changeit
:: import root CA certificate into server keystore

call keytool -import -alias server -keystore server.keystore -trustcacerts -file signed_server_cert.crt -storepass changeit
:: import signed server certificate into server keystore

Server code to create the SSL engine:

final KeyStore keyStore;
try (FileInputStream input = new FileInputStream(new File("server.keystore"))) {
  keyStore = KeyStore.getInstance("JKS");
  keyStore.load(input, "changeit".toCharArray());
 }
 final KeyManagerFactory factory = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
 factory.init(keyStore, "changeit".toCharArray());
 final SSLContext context = SSLContext.getInstance("TLS");
 context.init(factory.getKeyManagers(), null, null);
 final SSLEngine sslEngine = context.createSSLEngine();
 sslEngine.setUseClientMode(false);
 sslEngine.setNeedClientAuth(false);

And to create the SslContext on client side:

final SslContext sslContext = SslContextBuilder.forClient().trustManager(new File("truststore.pem")).build();

However, connection attempts from client to server result in the following exception:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "CN=CA, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" is not a CA certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at de.klenze_kk.chess.client.netty.MyLoggingHandler.channelRead(MyLoggingHandler.java:28)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "CN=CA, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" is not a CA certificate
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1339)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1550)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1396)
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1237)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1286)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
        ... 21 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: sun.security.validator.ValidatorException: TrustAnchor with subject "CN=CA, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" is not a CA certificate
        at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369)
        at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:263)
        at java.base/sun.security.validator.Validator.validate(Validator.java:264)
        at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
        at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317)
        ... 35 more
Caused by: sun.security.validator.ValidatorException: TrustAnchor with subject "CN=CA, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown" is not a CA certificate
        at java.base/sun.security.validator.PKIXValidator.verifyTrustAnchor(PKIXValidator.java:393)
        at java.base/sun.security.validator.PKIXValidator.toArray(PKIXValidator.java:333)
        at java.base/sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:366)
        ... 41 more

Can someone help, please? What am I doing wrong?

ssl-certificate ssl java
1 Answers

It looks like you are in the right direction but missing couple of essential parts when creating the CA and the Server certificates.

I would recommend to use the option genkeypair instead of genkey and you also need some extension to give your certificates some additional capabilities.

For the CA I would recommend:

-ext KeyUsage=digitalSignature,keyCertSign -ext BasicConstraints=ca:true,PathLen:3

And for the server I would recommend:

-ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -ext SubjectAlternativeName:c=DNS:localhost,IP:127.0.0.1

Next to that you need to make sure to temporally import the CA certificate into the Server certificate before replacing the unsigned server certificate with the signed one. This is something which is weird in my opinion but oracle/sun just designed it this way. In your steps you have done this correctly, but you can remove the CA certificate when you have imported the signed server certificate.

You also don't need the -trustcacerts option when importing the signed server certificate. And I would recommend to use importcert instead of import command when importing the signed server certificate.

So to sum up I would recommend to execute the following steps:

1. Create CA keystore

keytool -genkeypair -alias ca -keyalg RSA -keystore ca.keystore -storetype JKS -storepass changeit -keypass changeit -ext KeyUsage=digitalSignature,keyCertSign -ext BasicConstraints=ca:true,PathLen:3

2. Create server keystore

keytool -genkeypair -alias server -keyalg RSA -keystore server.keystore -storetype JKS -storepass changeit -keypass changeit -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -ext SubjectAlternativeName:c=DNS:localhost,IP:127.0.0.1

3. Create certificate signing request (CSR) for the server

keytool -certreq -alias server -keystore server.keystore -file server_signing_request.csr -storepass changeit -keyalg rsa

4. Sign the server CSR with the CA

keytool -gencert -infile server_signing_request.csr -outfile signed_server_cert.crt -keystore ca.keystore -alias ca -storepass changeit -ext KeyUsage=digitalSignature,dataEncipherment,keyEncipherment,keyAgreement -ext ExtendedKeyUsage=serverAuth,clientAuth -ext SubjectAlternativeName:c=DNS:localhost,IP:127.0.0.1 -rfc

5. Export the CA certificate

keytool -exportcert -rfc -alias ca -file truststore.pem -keystore ca.keystore -storepass changeit

6. Import the CA certificate into the server keystore

keytool -importcert -alias ca -keystore server.keystore -file truststore.pem -storepass changeit

7. Import the signed server certificate

keytool -importcert -alias server -keystore server.keystore -file signed_server_cert.crt -storepass changeit

8. Delete the imported CA certificate from the server keystore

keytool -delete -alias ca -keystore server.keystore -storepass changeit

Could you retry with these commands and share the results?

By the way I have written a tutorial here which might be helpful for you :) See: https://github.com/Hakky54/mutual-tls-ssl

user_67776950
Related