matplotlib tight_layout with annotation causes figure to get smaller and smaller

I have a FigureCanvasQTAgg that creates plots with the user selecting what data to plot, and the canvas updates the plots by calling self.fig.clear() and creating the new plot. This has worked fine but I ran into a problem. I have designed a certain plot with an annotation under the axes, but every time the figure is updated, the plot gets smaller and smaller. This code replicates the problem:

import sys
from PyQt5 import QtCore, QtWidgets
import matplotlib.pyplot as plt
from matplotlib.backends.backend_qt5agg import FigureCanvasQTAgg

class MplCanvas(FigureCanvasQTAgg):

    def __init__(self, parent=None, width=5, height=4, dpi=100):

        self.fig, self.axes = plt.subplots(figsize=(width, height), dpi=dpi, tight_layout=True)
        super(MplCanvas, self).__init__(self.fig)


    def make_plot(self, x, y, fmt='.k'):

        self.fig.clear()
        axes = self.fig.subplots()

        axes.plot(x, y, fmt)
        axes.invert_yaxis()
        axes.annotate('I make the plot get smaller and smaller', xy=(0,0), xycoords='figure points', fontsize='x-small', fontstyle='italic', annotation_clip=False)

        self.draw()

class MainWindow(QtWidgets.QMainWindow):

    def __init__(self, *args, **kwargs):
        super(MainWindow, self).__init__(*args, **kwargs)

        #Create a layout with central alignment
        self.layout1 = QtWidgets.QVBoxLayout()
        self.layout1.setAlignment(QtCore.Qt.AlignCenter)

        # Create a placeholder widget to hold our toolbar and canvas.
        self.widget1 = QtWidgets.QWidget()
        self.widget1.setLayout(self.layout1)
        self.setCentralWidget(self.widget1)

        #Create a matplotlib canvas and add it to the layout
        self.sc = MplCanvas(self, width=5, height=4, dpi=100)
        sizePolicy = QtWidgets.QSizePolicy(QtWidgets.QSizePolicy.Policy.Fixed, QtWidgets.QSizePolicy.Policy.Fixed)
        self.sc.setSizePolicy(sizePolicy)
        self.layout1.addWidget(self.sc)

        # Create a button
        self.button = QtWidgets.QPushButton()
        self.button.setText("Press me")
        self.layout1.addWidget(self.button)

        #Create the connection
        self.button.clicked.connect(self.plot_something)

        self.show()

    def plot_something(self):
        print('plotting')
        x = [0,1,2,3,4]
        y = [10,1,20,3,40]
        self.sc.make_plot(x, y)

app = QtWidgets.QApplication(sys.argv)
w = MainWindow()
app.exec_()

In this example, fiddling with the xycoords argument sometimes solves it, for example using xycoords='axes points' or 'data' seems to work. But this does not solve the issue in my actual application. Disabling tight layout also seems to solve it, but I need tight layout for various reasons.

The fact that the axes get smaller and smaller each time suggests that self.fig.clear() does not actually clear everything - something is being remembered between iterations of the plot. Is there a way I can completely purge the fig object and start a new one? Or would it be best to close the actual canvas and create a new one each time?