Loading a video dataset (Keras)

user_5175253 0
Viewed 2604

I'm trying to implement an LRCN/C(LSTM)RNN to classify emotions in videos. My dataset structure is split in two folders - "train_set" and "valid_set". When you open, either of them, you can find 3 folders, "positive", "negative" and "surprise". Lastly, each of these 3 folders has video-folders, each of which is a collection of frames of a video in .jpg. Videos have different length, hence a video-folder can have 200 frames, the one next to it 1200, 700...! To load the dataset I am using flow_from_directory. Here, I need a few clarifications:

  1. Will in my case flow_from_directory load the videos 1 by 1, sequentially? Their frames?
  2. If I load into batches, does flow_from_directory take a batch based on the sequential ordering of the images in a video?
  3. If I have video_1 folder of 5 images and video_2 folder of 3 videos, and a batch size of 7, will flow_from_directory end up selecting two batches of 5 and 3 videos or it will overlap the videos, taking all 5 images from the first folder + 2 of the second? Will it mix my videos?
  4. Is the dataset loading thread-safe? Worker one fetches video frames sequentially from folder 1, worker 2 from folder 2 etc... or each worker can takes frames from anywhere and any folder, which can spoil my sequential reading?
  5. If I enable shuffle, will it shuffle the order in which it would read the video folders or it will start fetching frames in random order from random folders?
  6. What does TimeDisributed layer do as from the documentation I cannot really imagine? What if I apply it to a CNN's dense layer or to each layer of a CNN?
keras tensorflow video python
1 Answers

  1. flow_from_directory is made for images, not movies. It will not understand your directory structure and will not create a "frames" dimension. You need your own generator (usually better to implement a keras.utils.Sequence)

  2. You can only load into batches if :

    • you load movies one by one due to their different lengths
    • you pad your videos with empty frames to make them all have the same length

  3. Same as 1.

  4. If you make your own generator implementing a keras.utils.Sequence(), the safety will be kept as long as your implementation knows what is each movie.

  5. It would shuffle images if you were loading images

  6. TimeDistributed allows data with an extra dimension at index 1. Example: a layer that usually takes (batch_size, ...other dims...) will take (batch_size, extra_dim, ...other dims...). This extra dimension may mean anything, not necessarily time, and it will remain untouched.

    • Recurrent layers don't need this (unless you really want an extra dimension there for unusual reasons), they already consider the index 1 as time.
    • CNNs will work exactly the same, for each image, but you can organize your data in the format (batch_size, video_frames, height, width, channels)
user_20972400
Related