Python text extraction from a video game screenshot

user_9894471 0
Viewed 1157

I am building a discord bot with discord.py for the video game Diablo 2. One of the functionalities requires the bot to extract the name and properties of items from Diablo 2 screenshots. I am currently using pytesseract for this but I am not getting sufficient results.

Example screenshot: enter image description here

I cropped the part of the item (the code needs to do this automatically later) and got this after preprocessing (see code below): enter image description here enter image description here

That is the code for preprocessing and extracting the manually cropped image:

def grayscale(image):
    return cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

def threshold(image):
    return cv2.threshold(image, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]

def dilate(image):
    kernel = np.ones((5,5),np.uint8)
    return cv2.dilate(image, kernel, iterations = 1)

image = cv2.imread('item.png')

scale = 10 
w = int(image.shape[1] * scale)
h = int(image.shape[0] * scale)
dim = (w, h)
image = cv2.resize(image, dim, interpolation = cv2.INTER_AREA)

image = grayscale(image)
image = threshold(image)
image = dilate(image)

custom_config = r'--oem 3 --psm 6 -c tessedit_char_whitelist=0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ%+'
print(pytesseract.image_to_string(image, config=custom_config))

It gives me these results:

sSPIRITWARD

WARD

ITamLavar66 Se a

Daryusa4if Oe

CHANCE Te BLece7a%

DURABILITY Blep5

REQUIREDSTRENGTHI76

RaguinaDLavat66

6%CHANCETSCASTLEVEL8FADEWHENSTRUCE

926%PFASTERBLeckRATE

29%IMCREASEDCHANCEOFBLeckiING2

Los t142%PNHANCEDDEFENSE a ALLRESISTANCES93i

Ay eYCeGlLOABSERE

ETHEREALCANNTBEREPAIRED

And I am unsure on how to proceed to get better results. The font and resolution are certainly difficult for OCR (you can see in the results how the OCR has problems with 5 and 6 especially). Here are some further pointers on the problem:

  • I do have the Diablo 2 font (Exocet) so I might be able to train my own model (?)
  • I do have list of all possible items and properties to further whitelist the results (however, I need the exact numbers for my bot's functionality)
  • I also tried another lib (keras-ocr) but did not get better results
python-tesseract screenshot opencv ocr python
1 Answers

I have a slightly improved solution

    1. Resize the image, so each character can be seen clearly
    1. Take each line one-by-one
Preprocessed Tesseract output
enter image description here oo : SPIRIT WARD © (
enter image description here _WARD '
enter image description here Irem LeveL: 88
enter image description here DeFEeNs@: 41°
enter image description here CHANCe Te@ BLeck: “73%
enter image description here DURABILITY: 51 @F 61
enter image description here RE@UIRED STRENGTH: 176
enter image description here ReouireD LeveL: 68 )
enter image description here 6% CHANCE T® CAST LEVEL 8 FADE WHEN STRUCK.
enter image description here 926% FasTER BLOCK RATE
enter image description here 29% INCREASED CHANCE @F BLOCKING
enter image description here 8h462% ENHANCED) DEFENSE ON
enter image description here ALL RESISTANCES @3h
enter image description here 9 COLD ABSORE
enter image description here ETHEREAL [CANN@T BE REPAIRED)

Comparison

Current result OP's result
oo : SPIRIT WARD © ( sSPIRITWARD
_WARD ' WARD
Irem LeveL: 88 ITamLavar66 Se a
DeFEeNs@: 41° Daryusa4if Oe
CHANCe Te@ BLeck: “73% CHANCE Te BLece7a%
DURABILITY: 51 @F 61 DURABILITY Blep5
RE@UIRED STRENGTH: 176 REQUIREDSTRENGTHI76
ReouireD LeveL: 68 ) RaguinaDLavat66
6% CHANCE T® CAST LEVEL 8 FADE WHEN STRUCK. 6%CHANCETSCASTLEVEL8FADEWHENSTRUCE
926% FasTER BLOCK RATE 926%PFASTERBLeckRATE
29% INCREASED CHANCE @F BLOCKING 29%IMCREASEDCHANCEOFBLeckiING2
8h462% ENHANCED) DEFENSE ON Los t142%PNHANCEDDEFENSE a ALLRESISTANCES93i
ALL RESISTANCES @3h Ay eYCeGlLOABSERE
9 COLD ABSORE ?
ETHEREAL [CANN@T BE REPAIRED) ETHEREALCANNTBEREPAIRED

I made a slight changes in the processing.

Code:

import cv2
import pytesseract

img = cv2.imread("YKEyQ.png")
(h, w) = img.shape[:2]
img = cv2.resize(img, (w*3, h*3))
gry = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
(h, w) = gry.shape[:2]

s_idx = 0
e_idx = int(h/15)

for i, _ in enumerate(range(0, 15)):
    gry_crp = gry[s_idx:e_idx, 0:w]
    thr = cv2.threshold(gry_crp, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)[1]
    thr = cv2.dilate(thr, None, iterations=1)
    cv2.imwrite("/Users/ahx/Desktop/res{}.png".format(i), thr)
    txt = pytesseract.image_to_string(thr, config="--psm 6")
    print(txt)
    s_idx = e_idx
    e_idx = s_idx + int(h/15)
    cv2.imshow("thr", thr)
    cv2.waitKey(0)
user_54042260
Related