Python OpenCV socket live video stream lagging

Question

I have created a chat application in python using sockets and wanted to implement a video calling feature.If I run the code for the server and client in the same machine then it is working perfectly fine. But as soon as I am shifting them to different machines the video is lagging. The rate of data transfer is too low. Below is the code of server and client machine respectively:

server.py

import socket
import cv2
import pickle

HEADER_SIZE = 10

server = socket.socket()
server.bind(("localhost", 1234))
server.listen(5)
client, addr = server.accept()

while True:
    full_msg = b''
    new_msg = True

    while True:
        msg = client.recv(4096)

        if new_msg:
            print(f"New message of length{msg[:HEADER_SIZE]}")
            msglen = int(msg[:HEADER_SIZE])
            new_msg = False

        full_msg += msg

        if len(full_msg) - HEADER_SIZE == msglen:
            print("Full msg received")

            frame = pickle.loads(full_msg[HEADER_SIZE:])
            cv2.imshow('server', frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

            new_msg = True
            full_msg = b''

    cv2.destroyAllWindows()
    break

client.py

import socket
import pickle
import cv2

client = socket.socket()
client.connect(("localhost", 1234))
cap = cv2.VideoCapture(0)

HEADER_SIZE = 10

while True:
    ret, frame = cap.read()

    data = pickle.dumps(frame)

    msg = bytes(f"{len(data):<{HEADER_SIZE}}", "utf-8") + data
    client.sendall(msg)

Does anybody know how to prevent this lagging and make a seamless video call? Thanks in advance.

user_1333724 · Answer

Would be mindful of a few things here:

Video streaming almost always has a buffer for frames being streamed in. If your server is accepting frames it should buffer them for a few seconds so that if network quality drops it can play from the buffer while waiting for new frames.
You're using images, not video. Video encoding works by sending keyframes every now and then and partial smaller frames between keyframes (there are also tons of optimizations in video codecs to deal with buffering, frames dropping, etc). If you're streaming video frame by frame as a full image your bandwidth use is going to be much much higher. You might want to use a real video streaming library (which will be harder)
Using pickle here might be adding overhead, you likely want to grab the raw bytes of the image and send them over the socket so that you can build the image without deserializing, although the two issues above are likely the larger performance issues.

With all that said, if you add a buffer (might be hard in python?) and don't pickle each frame you might see some improvement.

edit: it looks like this library might be useful for you use case: https://github.com/jeffbass/imagezmq

Python OpenCV socket live video stream lagging

1 Answers