Is there a way to capture the contents of a Tkinter canvas into an image?

Question

I am trying to build a handwritten digit recognition system, using convolutional neural networks. So far, I have developed a Tkinter window that contains a canvas, on which you can hand-write a digit. After I hit the 'Recognize' button, I want the program to take an image of that digit (i.e. the canvas) and run it through the neural network to predict the digit. The following is an implementation of some code I found online:

from tkinter import *
import win32gui
from PIL import ImageGrab, Image

cv = Canvas(width=400, height=400, bg='white', cursor='cross')
cv.grid(row=0, column=0, sticky=W)

hwnd = cv.winfo_id()
cv_rect = win32gui.GetWindoRect(hwnd)
img = ImageGrab.grab(cv_rect)

This code is able to grab an image. However, the image isn't of the canvas, but somewhere else on the screen, usually to the north-west of the canvas. The output looks something like this:

However, if I move the Tkinter window further away from the top left of my computer screen, the image doesn't even capture the top left of the Tkinter window, as in the above image.

For further perspective, here is the whole code of the file in which I am building this gui.

import numpy as np
from tkinter import *
from PIL import Image, ImageGrab
import win32gui
import tensorflow as tf
import matplotlib.pyplot as plt
import matplotlib
import win32api

cnn = tf.keras.models.load_model('\Users\hamza\Documents\VS Code\mnist.h5py')

#---------------------------------------------------------------
'''
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.mnist.load_data()

X_test = X_test.reshape(-1, 28, 28, 1)

X_test = X_test.astype('float32')

X_test /= 255

y_test = tf.keras.utils.to_categorical(y_test)

y_pred = cnn.predict(X_test)
print(y_pred[3700])
print(y_test[3700])

test_eval = cnn.evaluate(X_test, y_test, verbose=1)

print('Test loss:', test_eval[0])
print('Test accuracy:', test_eval[1])
'''

from pyscreenshot import grab
import pygetwindow as pgw
'''
im = grab(bbox=(500, 200, 800, 400))
im.show()
'''


def predict(img):
    #img = img.resize((28, 28))
    img = img.convert('L')
    img = np.array(img)
    #img.reshape(28, 28)
    plt.imshow(img, cmap=matplotlib.cm.binary, interpolation='nearest')
    plt.show()
    #img = img.reshape(1, 28, 28, 1)/255
    #pred = cnn.predict(img)[0]
    #print(pred)
    #print('---------------------')
    #return np.argmax(pred), max(pred)

def classify():
    '''
    hwnd = cv.winfo_id()
    print(hwnd)
    cv_rect = win32gui.GetWindowRect(hwnd)
    img = ImageGrab.grab(cv_rect)
    '''
    win = pgw.getWindowsWithTitle('tk')[0]
    winleft = win.left+9
    wintop = win.top+7
    winright = win.right-9
    winbottom = win.bottom-9
    final_rect = (winleft,wintop,winright,winbottom)
    img = ImageGrab.grab(final_rect)
    
    digit, accuracy = predict(img)
    print(digit)
    print(accuracy)
    #label.configure(str(digit) + ', ' + str(int(accuracy*100)) + '%')
        
def clear():
    cv.delete('all')
    
def draw(event):
    x = event.x
    y = event.y
    r = 8
    cv.create_oval(x-r, y-r, x+r, y+r, fill='black')

root = Tk()

cv = Canvas(width=400, height=400, bg='white', cursor='cross')
cv.grid(row=0, column=0, pady=2, sticky=W)

label = Label(text='Welcome', font=('Helvetica', 32))
label.grid(row=0, column=1, padx=2, pady=2)

clear_button = Button(text='Clear', command=clear)
clear_button.grid(row=1, column=0, pady=2)

recognize_button = Button(text='Recognize', command=classify)
recognize_button.grid(row=1, column=1, padx=2, pady=2)
        
cv.bind("", draw)

mainloop()

Someone, please help.

user_13382000 · Answer

I have used a similar functionality in my GUI and I used win32gui at first, but realized it was overkill for this small purpose, instead, just go for PyGetWindow and PIL.

Start by installing the module, by saying this in the terminal.

pip install PyGetWindow
pip install Pillow

then, this code will take a screenshot automatically after 3 seconds

from tkinter import *
from win32gui import FindWindow, GetWindowRect
import pygetwindow as gw
from PIL import ImageGrab

def ss():
    win = gw.getWindowsWithTitle('Trial')[0]
    winleft = win.left+9
    wintop = win.top+38 #change 38 to 7 to not capture the titlebar
    winright = win.right-9
    winbottom = win.bottom-9
    final_rect = (winleft,wintop,winright,winbottom)
    img = ImageGrab.grab(final_rect)
    img.save('Required Image.png')
#making the tkinter window
root = Tk()
root.title('Trial')

cv = Canvas(width=400, height=400, bg='white', cursor='cross')
cv.grid(row=0, column=0, sticky=W)

root.after(3000,ss)

root.mainloop()

Here, win = gw.getWindowsWithTitle('Title of the window')[0], you have to call your window name, like whatever the name of the window you set is, like, root.title('Hello World') then 'Title of the window' would be replaced by 'Hello World'. Or by default it will tkinter title will be 'tk'.

Or if your still reluctant on using win32gui then replace your ss() to,

def ss():
    win = FindWindow(None, 'Title of the window')
    rect = GetWindowRect(win)
    list_rect = list(rect)
    list_frame = [-9, -38, 9, 9]
    final_rect = tuple((map(lambda x,y:x-y,list_rect,list_frame))) #subtracting two lists

    img = ImageGrab.grab(bbox=final_rect)
    img.save('Image.png')

Extra Tip:

Why am i subtracting some amount from the pixels? its because, windows has decorations like drop shadow effect to the windows, which are also part of the windows and will be included in the screenshot, so i used this to get rid of those extra pixels,i guess it will make your neural network more efficient with no garbage items to look into.

Here is the image i got:

Hope this helped you, do let me know if any errors or doubts.

Extra reference - ImageGrab not grabbing precise bbox image

Cheers

Is there a way to capture the contents of a Tkinter canvas into an image?

1 Answers