How to keep a real-time track of how long two object intersect in a video?

user_5305512 0
Viewed 88

I am making a brushing app where I want to track how long a person brushes their teeth for. For that, I have built a toothbrush tracker, a mouth tracker, and also developed an algo to determine whether the toothbrush is inside the mouth (which I assume means the person is brushing).

Now, I just need to add a timer that can count real-time how long has the person been brushing for, and display it on the video feed. If the user pauses brushing, the timer should pause too; if the user resumes brushing, the timer should resume too. How do I make this?

My logic to detect when brushing is being done is the following:

  • extract the vertices of the toothbrush's bounding box
  • extract the left-most, top-most, right-most and bottom-most points of the mouth, and extract their coordinates
  • find whether these two rectangles overlap (which I am doing using a library called shapely - if you have a more primitive solution that doesn't need the use of a library, that will be great)
  • if they are overlapping, start the timer; if they aren't, pause the timer. This is the step I am not able to figure out how to do, and where I need help in.

This is roughly how my code looks like:

from shapely.geometry import Polygon
import cv2, face_recognition, time

brush_time = 0
video_capture = cv2.VideoCapture(0)

while True:
    ret, frame = video_capture.read()

    '''
    Code for extracting the toothbrush's coordinates (using Yolo),
    and the mouth's coordinates (using the face_recognition library)
    ...
    ...
    ...
    '''

    p1 = Polygon([(mouth_left_x, mouth_left_y), (mouth_top_x, mouth_top_y), (mouth_right_x, mouth_right_y), (mouth_bottom_x, mouth_bottom_y)])
    p2 = Polygon([(brush_top_left_x, brush_top_left_y), (brush_top_right_x, brush_top_right_y), (brush_bottom_left_x, brush_bottom_left_y), (brush_bottom_right_x, brush_bottom_right_y)])

    if p1.intersects(p2):
        
        ##############################################
        # This is where I want the timer logic to be #
        # start_time = time.time()                   #
        # end_time = ????                            #
        ##############################################

        brush_time = end_time - start_time
        text = 'Brushed teeth for {} seconds'.format(str(brush_time))
deep-learning face-recognition computer-vision opencv python-3.x
1 Answers

The answer will also contain some clean code advise, which you may ignore while implementing it on your end. But I would like to show you my preferred way of doing the same. So you will need 2 helper classes for this. The responsibility of the first class would be to give us a boolean value as to wheter the person is brushing or not. Try to encapsulate the overlapping rectangles logic etc within this class only. For the sake of simplicity I will be hard-coding a random brush-in and brush out duration.

class BrushDetector:
    def is_brushing(self, curr_frame_idx: int, frame) -> bool:
        """
        You provide the implementation here to detect whether the individual is holding brush inside his/her mouth.
        For sake of simplicity, I will simply hard-code the time when this shall return true in the attached video.
        """
        if 50 < curr_frame_idx < 100:
            return True
        if 150 < curr_frame_idx < 200:
            return True
        if 250 < curr_frame_idx < 300:
            return True
        return False

Now we build another class which shall record the brush-in and brush-out instances. The responsibility of this class would be to provide us with the total brush time. The logic for this class can be built as:

import datetime

class BrushRecorder:
    def __init__(self):
        self._start_time_ms = None
        self._total_time_ms = datetime.timedelta(0, 0, 0, 0, 0, 0, 0)

    def register_brush_in(self):
        if self._start_time_ms is None:
            self._start_time_ms = datetime.datetime.now()

    def register_brush_out(self):
        if self._start_time_ms is None:
            return

        delta = datetime.datetime.now() - self._start_time_ms
        self._total_time_ms += delta
        self._start_time_ms = None

    def get_brushing_time(self):
        if self._start_time_ms is None:
            return self._total_time_ms.microseconds / 1000
        else:
            return (self._total_time_ms + datetime.datetime.now() - self._start_time_ms).microseconds / 1000

Now we have build both the core classes. Lets get into the image reading part and try to integrate all the pieces together.

if __name__ == "__main__":
    video_stream = cv2.VideoCapture("path/to/stream/brush.mp4")
    recorder = BrushRecorder()
    detector = BrushDetector()
    frame_idx = 0
    while video_stream.isOpened():
        ret, frame = video_stream.read()
        if not ret:
            break

        if detector.is_brushing(frame_idx, frame):
            recorder.register_brush_in()
        else:
            recorder.register_brush_out()
        brushing_time = recorder.get_brushing_time()
        print(frame_idx, brushing_time)
        frame_idx += 1
    video_stream.release()

In the above snippet I have just logged the brush time to the console, but you can also use cv2.putText on each frame for GUI purposes or use this brushing_time value as required.

user_30519610
Related