Remove a graphs label with for loop in Python

I have a graph that displays this https://i.stack.imgur.com/rYzkv.jpg when a colored line (chemical element) is deleted, I want my corresponding annotated label to be deleted as well. I have it set up by appending the annotation labels to a list, if the label is not in any of the lines, delete the label that doesn't have that line showing.

The annotation list does in fact update to show the remaining labels after deleting the lines, but with tons of Attribute errors in between. And still the labels aren't being removed. How do I fix this using this idea?

P.S. The button that deletes the chemical line are in other files that I would rather not share. Over 20 files with different classes. So I hope to solve it using this method

def plot_element_peaks(peakClusters, x, hist, peakElementName):

annotation_list = []
minVal = get_bottom_val(hist)
ax = plt.gca()


for name, element in zip(peakElementName, peakClusters):

    color = next(ax._get_lines.prop_cycler)['color']

    biggestLine = element[0]
    for line in element[1:]:
        if line[1] > biggestLine[1]:
            biggestLine = line
    lineAmp = np.interp(biggestLine[0], x, hist) / biggestLine[1]

    for line in element:
        if lineAmp * line[1] > minVal:
            # this creates the element peak lines
            my_plot = plt.plot([line[0], line[0]], [minVal, lineAmp * line[1]], c=color)

            if line in biggestLine:
                name = ("+".join(map(str, name[:2])))

                my_labels = annotation_list.append(plt.annotate(name, xy=([line[0], lineAmp * line[1]]), textcoords="offset points",
                                   # how to position the text
                                   xytext=(0, 10),  # distance from text to points (x,y)
                                   c=color, size=8))

               
                for i in annotation_list:
                    if my_labels not in biggestline:     
                        try:
                            f.delete(i)
                            # possibly try to delete my_labels and create new plt.annotate with updated annotation list
                            print('removed element')
                        except AttributeError:
                            print('attribute error')