You don't have to use the from_list classmethod, that just exists for convenience, assuming a commonly used equal spacing between them.
You can convert your colors to the cdict required for the LinearSegmentedColormap. Starting with what you already have:
import matplotlib as mpl
colors = {
0.01: "#51158d",
0.25: "#5c2489",
0.5: "#693584",
1: "#9d9933",
3: "#e3dc56",
}
my_cmap = mpl.colors.LinearSegmentedColormap.from_list("my_cmap", list(colors.values()))

You basically need to do two things; normalize your range between 0 and 1, and convert the hex format to rgb.
For example:
from collections import defaultdict
cdict = defaultdict(list)
vmin = min(colors.keys())
vmax = max(colors.keys())
for k,v in colors.items():
k_norm = (k - vmin) / (vmax - vmin)
r,g,b = mpl.colors.to_rgb(v)
cdict["red"].append((k_norm, r, r))
cdict["green"].append((k_norm, g, g))
cdict["blue"].append((k_norm, b, b))
my_cmap = mpl.colors.LinearSegmentedColormap("my_cmap", cdict)

Note that the cdict requires the x-values to increase monotonically, which is the case with this example. Otherwise first sort the colors in the dictionary.
When using this to plot, you can/should use a norm to scale it between whatever values you want, for example:
norm = mpl.colors.Normalize(vmin=0.01, vmax=3)
Details about the format of the cdict can be found in the docstring:
https://matplotlib.org/stable/api/_as_gen/matplotlib.colors.LinearSegmentedColormap.html