I am trying to change the font type (from deja-vu to lets' say comic sans or calibri) for the text that appears along with matplotlib's annotate function. However, though I can change the font for all the other elements in the plot, it won't change it for the annotated text. Any help? Below is what I'm working with and how I globally set the font. You'll notice the difference in fonts when comparing the style of "1" on the plots. 
import numpy as np
import os, glob
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.font_manager
from matplotlib import rcParams
import pdb
matplotlib.font_manager._rebuild()
# THIS SETS ALL THE RELEVANT FONT STYLE FOR PLOTS
rcParams['font.family'] = 'sans-serif'
rcParams['font.weight'] = 'regular' #can omit this, it's the default
rcParams['font.sans-serif'] = ['calibri']
# test code
ax = plt.figure().add_subplot()
x,y = np.arange(10),np.arange(10)
ax.plot(x,y)
ax.annotate(r'$100^\circ$',
xy=(1.1, 0.1), xycoords='axes fraction',
xytext=(1.1, 0.8), textcoords='axes fraction',
arrowprops=dict(facecolor='black', shrink=0.01),
horizontalalignment='center', verticalalignment='top',
fontsize=15)