How is the best way to recognize frames with contains only white text on red gradient background?

user_13839770 0
Viewed 107

I'm capturing frames from video stream. In random moments, on constant place there is white text on the red gradient background. I want to get text from this frames (I use pytesseract) and save it to the database, so how can I detect that frames? When frames don't contains text, pytesseract returns senseless things and my code sends them to the database - this is unacceptable. Frames are cropped, so contains only white text on red background (only capital letters, one or two lines) or other random content. Sometimes can be possible situation when cropped is not good because the height of the red rectangle is reduced like this combination or unfortunately there is other text. In these cases pytesseract is weak.

import cv2
import pytesseract
import time
import mysql.connector
from difflib import SequenceMatcher

recent = ""
while True:
    cap = cv2.VideoCapture(VIDEO_URL)
    ret, frame = cap.read()
    img = cv2.cvtColor(frame, cv2.IMREAD_COLOR)[815:970, 360:1920] #crop frame
    text = pytesseract.image_to_string(img)
    if SequenceMatcher(None, recent, text).ratio() < 0.5: #save only when text is new
        sql = "INSERT INTO table(content, date) VALUES(%s ,%s)"
        val = (text, time.localtime())
        mycursor.execute(sql, val)
        db.commit()
        recent = text

How can I improve this solution and processing only croped frames with white text on red gradient background? Is something better then pytesseract for this solution?

cv2 image-processing ocr python
1 Answers

You can use color filtering to find a mask and then based on the percentage of area occupied by mask you can decide whether you need to extract the text of not. Modified code :

import cv2
import pytesseract
import time
import numpy as np
import mysql.connector
from difflib import SequenceMatcher

# bound for red colour
lower_red = np.array([0, 50, 50])
upper_red = np.array([10, 255, 255])

threshold = 35
recent = ""
while True:
    cap = cv2.VideoCapture(VIDEO_URL)
    ret, frame = cap.read()
    img = cv2.cvtColor(frame, cv2.IMREAD_COLOR)[815:970, 360:1920] #crop frame

    h, w = img.shape[:2]

    # converting to hsv
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    
    # filtering 
    mask = cv2.inRange(hsv, lower_red, upper_red)
    
    # percentage of area which is red
    total = h * w
    count = cv2.countNonZero(mask)
    percentage = (100 * count) / total
    
    if SequenceMatcher(None, recent, text).ratio() < 0.5 and percentage >= threshold: #save only when text is new
        custom_config = r'--oem 3 --psm 6'
        text = pytesseract.image_to_string(mask, config=custom_config)
        sql = "INSERT INTO table(content, date) VALUES(%s ,%s)"
        val = (text, time.localtime())
        mycursor.execute(sql, val)
        db.commit()
        recent = text

Also you can improve this by finding the largest connected component in mask and considering that part only

user_100371500
Related