how to plot arrows to point to cells inside a table in matplot

I want to achieve something like this: needleman matrix

I have tried to plot an arrow using this function: plt.arrow(0.02, -0.02, -0.002, 0.002, width=0.0005)

but the problem is that when the table gets bigger, its position change with respect to the X and Y axis, so i need a method to get the arrow to point to the cells themselves, rather than x and y coordinates.

Full code:

fig, ax = plt.subplots()
fig.patch.set_visible(False)
#ax.axis('off')
ax.axis('tight')
df = pd.DataFrame(matrix, columns=list(" " + sequence_a))

colors = [['w'] * N for _ in range(M)]

# COLOR THE MAP
x = [j for sub in queues for j in sub]
for i in range(M):
    for j in range(N):
        if (i, j) in x:
            colors[i][j] = 'g'

ax.table(cellText=df.values, colLabels=df.columns,
         loc='center', rowLabels=" " + sequence_b,
         cellColours=colors, colWidths=[0.05 for x in df.columns], )
plt.arrow(0.02, -0.02, -0.002, 0.002, width=0.0005)
fig.tight_layout()
plt.show()

where matrix is the list of lists containing the numbers (table), I also have a dictionary that have the arrows data(which cell indices pointing to which)

Full project github link: https://github.com/YoussefEmad99/Needleman-Wunsch-Bioinformatics/blob/main/Needleman-Wunsch-Algorithm.py