matplotlib imshow -- use any vector as axis

I would like to use any vector as an axis in plt.imshow().

A = np.random.rand(4, 4)
x = np.array([1, 2, 3, 8])
y = np.array([-1, 0, 2, 3])

I imagine something like this:

plt.imshow(a, x_ax=x, y_ax=y)

I know there is an extent parameter available, but sadly it does not allow for non-equally spaced vectors.

Can anyone please help? Thanks in advance.