I'm currently going thru the paper CondInst (Condition Instance Segmentation https://arxiv.org/pdf/2003.05664.pdf) and they use dynamic filters as conv layers. I saw there implementation in pytorch but I couldn't implement it with keras because the tf.nn.conv2d function takes a batch as input but can only apply one filter onto that whole batch. However, for the dynamic filter to work, it requires the each sample applied by the conv2d parameters it generates by itself.
A snippet of my code are below:
num_of_conv_in_controller = 4
mask_controllers = ... Some dense layer to get the parameter.
mask_controllers = Reshape((3, 3, 8, 8, num_of_conv_in_controller))(mask_controllers)
input = ... Some input tensor with shape (None, W, H, 8) # here I use 8 as number of channels for simplicity
for i in range(num_of_conv_in_controller):
kernel = mask_controllers[..., i]
input = tf.nn.conv2d(input, kernel, [1, 1, 1, 1], "SAME")
However, this will raise the error that kernel can only take rank 4 tensor because the current kernel shape is actually (None, 3, 3, 8, 8). The actual correct way should be looping over the mask_controllers by the first dimension and apply each kernel individually. But after some research i couldn't find any way to loop over a None dimension.
Currently I hacked it in the way that i passed in my batch size into the model and then reshaped all the dimension to the batch_size but then the model can only works with fixed batch size of input which cause issue in inference time. I'm not sure if any one has tried implementing this.