Something wrong when computing the receptive field using non-zero gradient in Keras

I'm trying to compute the receptive field of some specific neurons based on the non-zero gradient but found one strange thing.

The following is a simple NN model built in keras. The remaining parts are to calculate the gradient of the output (here the targeted neuron of which pos is (0,2) on the first channel) of conv2d_4 w.r.t its input. Through finding those non-zero values on the gradient map, we can easily locate the receptive field of one neuron. The ideal receptive field of one neuron in the output of conv2d_4 w.r.t its input should be 3x3 since the kernel size of conv2d_4 is 3x3, but the non-zero gradient map is a 4x5 patch (given by those TRUE values in f_sum).

import numpy as np
import keras.backend as K
import matplotlib.pyplot as plt
from keras.models import load_model, Model
from keras.layers import Dense, Dropout, Flatten, Conv2D, MaxPooling2D,Input, AveragePooling2D, Lambda

def model_build_func(input_shape=(25,25,1)):
    inp = Input(shape=input_shape, name='input')

    x = Conv2D(32, (3,3), activation='linear', name='conv2d_1')(inp)
    x = Conv2D(32, (3,3), activation='linear', name='conv2d_2')(x)
    x = AveragePooling2D(pool_size=(2,2))(x)

    x = Conv2D(64, (3,3), activation='linear', name='conv2d_3')(x)
    x = Conv2D(64, (3,3), activation='linear', name='conv2d_4')(x)
    x = AveragePooling2D(pool_size=(2,2))(x)

    x = Flatten()(x)
    x = Dense(units=64, name='dense_1')(x)
    x = Dense(units=2, name='dense_2')(x)

    model = Model(inputs=inp, outputs=x)
    return model

# used for building the Lambda layer
def get_mask_tensor(input_tensors, x_pos, y_pos, channel_idx):
    mask_tensor = K.tf.gradients(input_tensors[0][:,x_pos,y_pos,channel_idx], input_tensors[1])[0]
    return mask_tensor

#specify the position of the neuron that we want to compute the RF
x_pos = 0
y_pos = 2
channel_idx = 0

layer_idx = 5 # the layer: conv2d_4

model = model_build_func()
current_layer = model.layers[layer_idx]

#get the gradient tensor
mask_tensor = Lambda(get_mask_tensor, output_shape=K.int_shape(model.input),
                     arguments={'x_pos':x_pos, 'y_pos':y_pos, 'channel_idx':channel_idx})([current_layer.output, current_layer.input])

#create a keras model
new_model = Model(inputs=[model.input], outputs=[mask_tensor])

#get the value of the gradient map
gradient_map = new_model.predict(0.1*(np.random.random(size=(32,25,25,1))-0.05))
f_sum = np.sum(np.abs(gradient_map), axis=-1)
f_sum = np.sum(np.abs(f_sum), axis=0)

#f_sum is a binary array. 
#It should has a 3x3 patch with TRUE values, but here it's 4x5
plt.imshow(f_sum!=0)
plt.grid()
plt.show()