python image streaming from a buffer optimization

Viewed 77

I am working on Python + flask in an embedded device with limited computing power.

I would like to stream a video, using images (RGBa) stored on the device as a pixels array. Currently, I am able to receive the video but the overall performance is not sufficient to get a smooth output.

Here is the current optimization I already did :

  • reduce the number of copies by reading into the memory region
  • remove the color alpha since it is not used.
  • reduce by 4 the size of the image
    from PIL import Image

    @bp.route("/stream")
    def stream():
        def loop():
            yield b"--frame\r\n"
            while True:
               raw_data = view[ frame_id * BUF_SIZE : (frame_id + 1) * BUF_SIZE ]
               image_np = np.frombuffer(np.asarray(raw_stream[::]),
                          dtype=np.uint8
                          ).reshape((600, 1024, 4))
               img_byte_arr = io.BytesIO()
               jpeg_img = Image.fromarray(image_np[::4, ::4, :3], mode="RGB")
               jpeg_img.save(img_byte_arr, format="JPEG")
               yield (
                  b"Content-Type: image/jpeg\r\n\r\n"
                  + img_byte_arr.getvalue()
                  + b"\r\n--frame\r\n"
               )
        return Response(loop(), mimetype="multipart/x-mixed-replace; boundary=frame")

Based on some mesurement it seems that the operation Image.fromarray(image_np[::4, ::4, :3], mode="RGB") is the bottleneck since it tooks up to 0.2sec to be executed (10FPS).

Is there another way of converting an array to an image (JPEG or whatever format) that could speed up the process? Maybe another library than PIL?

1 Answers

If you have access to GPUs, I would recommend to use PyTorch using cuda to save and load your images. The transformations you make such as reducing format are also parallelized on GPU. The speedup may be significant, and if it is still not enough you can use a dataloader to not keep in memory all of your images.

Related