Detecting a physical object, reading text from it, with OpenCV + Tesseract, then ignoring it in future

So I've got image processing down, and I can successfully read text from a type of trading card. My problem is that I am reading frames, so it's reading the same card basically over and over again.

Originally I was thinking of getting it to read the card, remember the text, then if the text is matching discard it. Problem is as I pull it away it reads the text slightly differently.

Is it possible to basically track the entire card as an 'object' of some kind, so it knows that the card it's currently viewing is card X, and to ignore that until card Y appears.

Here is the code I am using to grab the frames

while True:
    ret, frame = cap.read()
    cv2.imshow("Frame", frame)

    process_card(frame)

    key = cv2.waitKey(1)
    if key == ord("q"):
        break

process_card(frame) literally just grabs the frame then returns the text, it's just some simple preprocessing and it gets the text just fine. The text is just a giant string, but I could break it up to just the titles of the cards, if that'd help.