(Differentiable Image Sampling) Custom Integer Sampling Kernel, Spatial Transformer Network

user_12833047 0
Viewed 436

I was looking through the Spatial Transformer Network paper, and I am trying to implement a custom grid_sample function (inheriting the autograd.Function class) in PyTorch for the Integer Sampling Kernel.

While defining the backward function, I have come across the following conundrum.

Given that the integer sampling works as the following:

enter image description here

I think that the gradients w.r.t the input map and the transformed grid (x_i^s, y_i^s) should be like the following:

Gradient w.r.t. input map: enter image description here

Gradient w.r.t transformed grid (x_i^s): enter image description here

Gradient w.r.t transformed grid (y_i^s): enter image description here

as the derivative of the Kronecker delta function is zero (I'm unsure about this!! - HELP)

Derivative of the Kronecker delta?

enter image description here

Thus I am reaching a conclusion that the gradient w.r.t to the input should be: a tensor of the same size as the input filled with ones if the pixel was sampled and 0 if it wasn't sampled, and the gradient w.r.t the transformed grid should be a tensor full of zeros.

However, if the gradient of the transformed grid is 0, then due to the chain rule, no information will be passed on to the layers before the integer sampler. Therefore I think the derivate with respect to the grid should be something else. Could anybody point out what I'm doing wrong?

Many thanks in advance!

pytorch deep-learning math image-processing
1 Answers

For future reference, and for those who might have had similar questions to the one I posted.

I've emailed Dr Jaderberg (one of the authors of the 'Spatial Transformer Networks') about this question and he has confirmed: "that the gradient wrt the coordinates for integer sampling is 0.". So I wasn't doing anything wrong, and it was right all along!

He was very kind in his response and expressed that integer sampling was mentioned in the paper to introduce the bilinear sampling scheme, and have given insights into how to possibly implement integer sampling if I really wanted to:

"you could think about using some numerical differentiation techniques (e.g. look at difference of x to its neighbours). This would assume smoothness in the image wrt coordinates."

So with great thanks to Dr Jaderberg, I'm happy to close this question.

I guess thinking about how I'd use numerical methods to implement the integer kernel for the sampling function is another challenge for myself, but until then I guess the bilinear sampler is my friend! :)

user_128330470
Related