Why is my annotation causing my figure size to change in matplotlib?

I am quite new to matplotlib and I am trying to do something simple, but I can't quite get the hang of it.

So I'm trying to piece together a couple of png images and label them A, B, C etc... but I want the labels to be in terms of figure fractions, so I can scale them in the future.

Here's my code for plotting the images:

im, h, w = loadImList(imList)

total_height = h[0] + h[1]
total_width = max(w[0],w[1])

new_img = np.ones(shape=(total_height, total_width, 3))

hStart[0] = 0
hStart[1] = h[0]
wStart[0] = 0
wStart[1] = 0

nIm = len(im)
for iDx in np.arange(len(im)):
    new_img[np.int32(hStart[iDx]):np.int32(hStart[iDx]+h[iDx]),np.int32(wStart[iDx]):np.int32(wStart[iDx]+w[iDx])]=im[iDx]


fig_w = 4.7
fig_h = total_height / total_width * fig_w

fig, ax = plt.subplots(figsize=(fig_w, fig_h), constrained_layout=True)
fig.set_facecolor("white")
plt.imshow(new_img)

ax.set_xticks([])
ax.set_yticks([])

And this is what I get:

And then I tried to write a function that adds in the annotations:

def addTextbox(textstr, x, y, ax):
    offsetbox = TextArea(textstr)
    ab = AnnotationBbox(offsetbox, (x,y),
                    xybox=(x, y),
                    xycoords='figure fraction',
                    boxcoords="figure fraction",
                    box_alignment=(0, 0),
                    frameon=False,
                    pad=0,
                    arrowprops=None,
                    fontsize=36,
                    clip_on = False)
    ax.add_artist(ab)
    return ax

Adding ax = addTextbox("B)", 0.05, 0.5, ax) gives the image below which is reasonable.

But, when I do

ax = addTextbox("A)", 0.01, 0.6, ax)
ax = addTextbox("B)", 0.05, 0.5, ax)

I get a white margin on the left side of the image, and the B) annotation shifts. I don't understand why this is happening, or how do I solve it. Any help will be appreciated. Thanks!