How to feed image sequences to convolutional layers and apply conv-lstm cells after?

user_10863265 0
Viewed 451

I'm currently trying to implement the following paper: https://research.nvidia.com/sites/default/files/publications/dnn_denoise_author.pdf

I have troubles in adapting my network, which currently processes only single images, to processing image sequences. My data has the following shape: (7, 512, 512, 1), where 7 is the number of frames in my sequence, 512 is the width and height of an image and 1 the number of channels.

My question is how to pass a sequence through convolutional layers? (the conv3d suggestion that I saw for other questions of this type seems weird, since I have 7 frames). Then, I wish to pass the result of the convolutional layers to a ConvLSTM block, however, is this even possible given the feature maps obtained after convolutions and maxpooling? (Other answers for using ConvLSTM block refer only to applying them on the sequence directly). The result of this operations will be again fed to convolutions and maxpooling and so on.

I have also checked the other questions involving CNN and RNN and I was thinking of using the TimeDistributed(...(...)) type of functions, but I am not sure if I'm going in the right direction. Any piece of advice is more than welcomed.

Thank you for your time!

recurrent-neural-network lstm sequence image python
1 Answers

I am facing similar situation where I have n frame sequence and I would like to predict the next frame after given sequence, the solution would be to forward image n through network and get loss from frame n+1 and repeat(n+1, n+2 etc.). Hope I understand this right.

user_58768190
Related