Efficient spatially varying convolution in python

I'm trying to find an efficient way to convolve a 2D numpy array with a bunch of filters at different locations. That is: I have an array A(h x w) and a list of kernels K[k1, k2, k3, ...] and a kernel selector map S(h x w). Now, I want to perform a spatially varying convolution, where each pixel at A[i, j] is convolved with kernel K[S[i, j]]. Is there an existing function for that or do I have to Cythonize it? Opencv and scipy only have uniform convolution. One inefficient solution is to convolve A with every kernel in K and use S to select. But that's a ton of wasted computation and not practical with large kernel size.