Fixing OpenCV Python stream ingestion

user_3019008 0
Viewed 82

I'm trying to break a video down into frames, draw some boxes and labels on the frames, and then feed them into a RedisImageStream, like this:

   def get_last(self):
        ''' Gets latest from camera and model '''
        p = self.conn.pipeline()
        p.xrevrange(self.camera, count=1)  # Latest frame
        p.xrevrange(self.boxes, count=1)   # Latest boxes
        cmsg, bmsg = p.execute()
        if cmsg:
            last_id = cmsg[0][0].decode('utf-8')
            label = f'{self.camera}:{last_id}'
            data = io.BytesIO(cmsg[0][1][self.field])
            img = Image.open(data)
            if bmsg:
                boxes = np.fromstring(bmsg[0][1]['boxes'.encode('utf-8')][1:-1], sep=',')
                label += ' people: {}'.format(bmsg[0][1]['people'.encode('utf-8')].decode('utf-8'))
                for box in range(int(bmsg[0][1]['people'.encode('utf-8')])):  # Draw boxes
                    x1 = boxes[box*4]
                    y1 = boxes[box*4+1]
                    x2 = boxes[box*4+2]
                    y2 = boxes[box*4+3]
                    draw = ImageDraw.Draw(img)
                    draw.rectangle(((x1, y1), (x2, y2)), width=5, outline='red')
            arr = np.array(img)
            arr = cv2.cvtColor(arr, cv2.COLOR_BGR2RGB)
            cv2.putText(arr, label, (10, 40), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (0,0,255), 1, cv2.LINE_AA)
            ret, img = cv2.imencode('.jpg', arr)
            return img.tobytes()

At this point, I should be getting bytes. The output of get_last feeds into this generator:

def gen(stream):
    while True:
        try:
            frame = stream.get_last()
            yield (b'--frame\r\n'
                b'Pragma-directive: no-cache\r\n'
                b'Cache-directive: no-cache\r\n'
                b'Cache-control: no-cache\r\n'
                b'Pragma: no-cache\r\n'
                b'Expires: 0\r\n'
                b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')
        except Exception as exception:
            # Output unexpected Exceptions.
            logging.error("Error occurred", exc_info=True)

The output of the generator should go into a return stream on an HTTP route:

@app.route('/video')
def video_feed():
    return Response(gen(RedisImageStream(conn, args)),
                    mimetype='multipart/x-mixed-replace; boundary=frame')

Instead, this throws an error per frame:

in gen
b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')
TypeError: cannot convert 'NoneType' object to bytes

How should I debug this?

opencv python
1 Answers

In get_last() you have

 if cmsg:
      # ... code ...
      return frame

but you don't have

 else:
      # ... code ...
      return frame

so sometimes it can runs return one at the end of this function

and then frame = stream.get_last() may get frame = None

You should check it before sending frame

    frame = stream.get_last()
    #if frame is not None:
    if frame:
        yield (b'--frame\r\n'
            b'Pragma-directive: no-cache\r\n'
            b'Cache-directive: no-cache\r\n'
            b'Cache-control: no-cache\r\n'
            b'Pragma: no-cache\r\n'
            b'Expires: 0\r\n'
            b'Content-Type: image/jpeg\r\n\r\n' + frame + b'\r\n\r\n')

Sometimes camera may need time for warm up (ie. probably in Raspberry Pi) and then it may send None instead of frame - and you have to check if frame is not None.

As for debuging.

I saw problem with None in frame many times so already knew where to search problem but if you want to debug it then you could use print() to check values in variable and which part of code is executed - it is called "print debuging".

First you should check print(frame, type(frame)) before line which gives error. I expect you get None. Because you get frame from get_last() so next you should use print() inside get_frame() to see what you have in variables and which part of code is executed. And if you would put print("I am inside 'if'") inside if cmsg: then you shouldn't see this text on screen. And this should give you clue where is problem. You could check print(cmsg, type(cmsg)) to see what you get in this variable. Because you get this value from p.execute() then you should check code in p.execute() or p. And this way you could move back in code until you place which starts this problem.

user_18320580
Related