Transmission times fluctuating when using TLS 1.2 via OpenSSL

I used TLS 1.2 via OpenSSL 1.0.2 to transmit data. When sending small packets, I've encountered a problem. The transmission time of each of the small packets with 25 bytes user data is very low. With times below 1ms everything works in the expected range.

The problem which I encounter is shown by the attached picture

It shows two different runs of my test setup. The transmission time suddenly increases by more than 40ms (The next packet after the highlighted one). This is obviously caused by a delayed sending of the user data. I wrote a minimal demonstration server to demonstrate the problem. The weird thing, which makes me think, is the fact that always the same user data telegram is influenced. Each transmitted packet starts with the current user telegram number (in the picture 6764 at the very bottom). This is more than interesting and makes the common causes, like packet drop or process suspension a very unlikely cause of this problem. (The server process is nevertheless started with nice --20 just to be sure.) This would cause random telegram numbers to be influenced.

while (1)
{
    int32_t recv_size = SSL_read(connectionSocket, buffer, sizeof(buffer) - 1);
    if (recv_size < 0)
    {
        std::cout << "Could not receive data\n";
        break;
    }
    else if (0 == recv_size)
    {
        std::cout << "Client closed connection\n";
        break;
    }
    /* Send data */
    if (SSL_write(connectionSocket, buffer, recv_size) != static_cast<int32_t>(recv_size))
   {
        std::cout << "Could not send data\n";
        break;
    }
}