Tls 1.3 client does not report failed handshake when client certificate verification by server failed

user_2263489 0
Viewed 1204

I have a C client using OpenSSL that is failing a test when using a certificate that fails validation on the server side during the SSL_do_handshake() call on the server. When the application was using TLS 1.2 The SSL_do_handshake() failure on the server would be reported back to the client when it called SSL_do_handshake() as a failure return value.

When upgrading my application to OpenSSL 1.1.1 and TLS 1.3 I noted that while the validation error is still occurring on the server, it was no longer being reported back to the client.

I'm aware that the handshake protocol got completely re-written as part of TLS 1.3 however it seems like with all of the various callbacks available I should be able somehow on the client side to determine that authentication has failed without having to attempt to write data to the server.

Has anyone else encountered this and can they recommend a path forward?

tls1.3 openssl ssl
1 Answers

The server and client in both TLSv1.2 and TLSv1.3 consider the handshake to be complete when they have both written a "Finished" message, and received one from the peer. This is what the handshake looks like in TLSv1.2 (taken from RFC5246):

      Client                                               Server

      ClientHello                  -------->
                                                      ServerHello
                                                     Certificate*
                                               ServerKeyExchange*
                                              CertificateRequest*
                                   <--------      ServerHelloDone
      Certificate*
      ClientKeyExchange
      CertificateVerify*
      [ChangeCipherSpec]
      Finished                     -------->
                                               [ChangeCipherSpec]
                                   <--------             Finished
      Application Data             <------->     Application Data

So here you can see that the client sends its Certificate and Finished messages in its second flight of communication with the server. It then waits to receive the ChangeCipherSpec and Finished messages back from the server before it considers the handshake "complete" and it can start sending application data.

This is the equivalent flow for TLSv1.3 taken from RFC8446:

       Client                                           Server

Key  ^ ClientHello
Exch | + key_share*
     | + signature_algorithms*
     | + psk_key_exchange_modes*
     v + pre_shared_key*       -------->
                                                  ServerHello  ^ Key
                                                 + key_share*  | Exch
                                            + pre_shared_key*  v
                                        {EncryptedExtensions}  ^  Server
                                        {CertificateRequest*}  v  Params
                                               {Certificate*}  ^
                                         {CertificateVerify*}  | Auth
                                                   {Finished}  v
                               <--------  [Application Data*]
     ^ {Certificate*}
Auth | {CertificateVerify*}
     v {Finished}              -------->
       [Application Data]      <------->  [Application Data]

One of the advantages of TLSv1.3 is that it speeds up the time taken to complete a handshake. In TLSv1.3 the client receives the "Finished" message from the server before it sends its Certificate and Finished messages back. By the time the client sends its "Finished" message, it has already received the "Finished" and so the handshake has completed and it can immediately start sending application data.

This of course means that the client won't know whether the server has accepted the certificate or not until it next reads data from the server. If it has been rejected then the next thing the client will read will be a failure alert (otherwise it will be normal application data).

I'm aware that the handshake protocol got completely re-written as part of TLS 1.3 however it seems like with all of the various callbacks available I should be able somehow on the client side to determine that authentication has failed without having to attempt to write data to the server.

It's not writing data to the server that is important - it is reading data. Only then will you know whether the server has sent an alert or just normal application data. Until that data has been read there are no callbacks available in OpenSSL that will tell you this - because OpenSSL itself does not know due to the underlying protocol.

user_19466790
Related