My DTLS server when compiled with either OpenSSL 3.0.2 or OpenSSL 1.1.1f, after sending out Server Hello, is unable to parse further SSL records from Client. Following is the error in OpenSSL error stack error:0A000119:SSL routines::decryption failed or bad record mac.
Same server code when I compile with OpenSSL 1.0.2g, handshake is always successful. But because of unrelated requirement, I need to use OpenSSL 1.1.1+ . The following is a simplified snippet of my server code:
SSL_CTX *s_ctx = SSL_CTX_new(DTLSv1_server_method());
SSL *ssl = SSL_new(s_ctx);
SSL_set_accept_state(ssl);
BIO* read_bio = BIO_new(BIO_s_mem());
BIO_set_mem_eof_return(read_bio, -1); //similarly for write_bio ..
SSL_set_bio(ssl, read_bio, write_bio);
unsigned char buffer[DTLS_PACKET_MAX_SIZE]; //DTLS_PACKET_MAX_SIZE is 8192 + 32
while(1)
{
memset(&buffer, 0, DTLS_PACKET_MAX_SIZE);
int length = recvfrom(fd, buffer, DTLS_PACKET_MAX_SIZE-1 , 0, (struct sockaddr *)&client_addr, &addrlen); //buffer gets populated with data
int buffered_bytes = BIO_write(read_bio, buffer, len); //data in buffer is copied to bio
memset(&buffer, 0, DTLS_PACKET_MAX_SIZE);//reused buffer for SSL_read
int len = SSL_read(ssl, buffer, buffered_bytes); //'should' decrypt data in bio into SSL records.
int err_stack_code = 0;
while ((err_stack_code = ERR_get_error()) != 0)
{
my_stderr_debug_macro("SSL handshake error: %s", ERR_error_string(err_stack_code, NULL));
}
}
The tests using OpenSSL 1.0.2g, 1.1.1f and 3.0.2 used the same certificate-key pairs and server codes, so I don't think certificates are the issue (?). I am also not sure what is the cause of corruption of MAC here, because I am resetting the buffer before and after use, so I am lost on how to debug further.
The wireshark screenshot of failed scenario: Server compiled with OpenSSL 3.0.2