JRTPLib receiving data from multiple clients concurrently causes a slowdown

I have a receive session on server app and two clients sending their audio data to that server.

On server app so far I simply tried these two examples: https://github.com/j0r1/JRTPLIB/blob/master/examples/example4.cpp https://github.com/j0r1/JRTPLIB/blob/master/examples/example2.cpp

which boils down to this code to receive the data in separate thread:

 session.BeginDataAccess();
    if (session.GotoFirstSource())
    {
        do
        {
            RTPPacket *packet;

            while ((packet = session.GetNextPacket()) != 0)
            {
                //access to packet here
                session.DeletePacket(packet);
            }
        } while (session.GotoNextSource());
    }
    session.EndDataAccess();

For one client sending it works fine, but when it's two clients sending it looks like the receive session can't keep up. It's not the whole server app getting locked tho, I can request the client list and TCP thread responds during this slowdown.

I'm very new to using RTP, forgive me If im misunderstanding the example code and I will be very thankful for any tip I can get on how to solve this.

So I'm trying to receive data from multiple client apps in one receiver session, but for some reason it doesn't work too fast. What could be the reason? RTP session is designed to get data from multiple sources iirc.