I have a question that strikes me as being fairly simple but I'm having a hard time wrapping my head around it.
I'd like to use built in data modules in both Tensorflow and Keras to load in images for both training & validation sets. Right now the data is saved as an .npy file.
The key thing I'd like to do is not load all of the images in from memory, and I'd like to do it locally.
A lot of the examples I'm seeing are either using directory structures different from mine, are loading in all of the Numpy files at once using normal file I/O, or making use of the tf.keras.utils.get_file command which only downloads files remotely.
Here's a quick example of what I'd like to do.
In a google cola I'm running the following code:
import tensorflow as tf
import numpy as np
from tensorflow import keras
(X_train, y_train), (X_test, y_test) = keras.datasets.mnist.load_data()
# I repeat this command for each of x_train, x_test, y_train, y_test
with open('x_test.npy', 'wb') as outfile:
np.save(outfile, X_test)
In google colab, my folder directory looks like this:
content/
-- x_train.npy
-- x_test.npy
-- y_train.npy
-- y_test.npy
It seems like there's something simple I'm missing. Maybe I'm just not synthesizing the documentation like I should, but from these files I'd like to use existing keras/tensorflow functionality to have a data pipeline that will not load in all of the data at once and get validation scores after each epoch while training.
thank you.