Retrieve actual data from raster with colorbar Python

I know that matplotlib can map data on RGB values. A classic exemple could be values of elevation mapped with the "terrain" cmap from matplotlib. Captured from GRASS tutorial

I have an RGB tiff file corresponding to magnetic anomalies. This is useless because there is actually no data but only RGB values. But... If only I could "reverse", "un-normalize" the RGB values from the colobar to retrieve the data... That would be amazing! Is there a function I do not know in matplotlib (or anything else) to do that? Like:

def fictionnal_fonction(img=TiffFile, min=min_on_colorbar, max=max_on_colorbar, colormap=viridis):

return greyscale_array_actual_data

Thank you very much!!!