I have around 2.1 million multi-channel images stored individually as npy-file in a directory. The directory currently works as centralized storage for all images. The images with 25 Pixels x 25 Pixels x 10 Channels belong to a time series (multiple pictures for one day, >10-year horizon):
main_directory/
...2010_01_01_NY.npy
...2010_01_01_SF.npy
...2010_01_01_LA.npy
...2010_01_01_BER.npy
...2010_01_02_NY.npy
...2010_01_02_SF.npy
...
Ideally, I would like to use a sliding window and move forward year by year. To avoid storing a lot of duplicate data (the files are already 60+GB), I try to create a new directory with links to the files to create a TensorFlow Dataset using the Directory Structure (e.g. image_dataset_from_directory).
main_directory/
...Class1/
......2010_01_01_NY.npy(Link)
......2010_01_02_NY.npy(Link)
......2010_01_01_SF.npy(Link)
...Class2/
......2010_01_01_BER.npy(Link)
......2010_01_02_SF.npy(Link)
......2010_01_01_LA.npy(Link)
...
I tried using image_dataset_from_directory but that did not accept the npy-files or links. Does anyone know how to implement that, or is there a more efficient way?