How do you flow_from_directory in keras when your input does not consist in images?

user_12607108 0
Viewed 320

Is there an equivalent to flow_from_directory that works on non-image data? More precisely, I have my data saved as a pickle file consisting of a numpy array. The data is already in the format that I need for the training of the network; is there a way to fit my model by reading each row vector of the pickle file? Or alternatively, saving my row vectors as single pickle files (or other formats) and fitting the model reading them one by one?

keras pickle python
1 Answers

Yes you can use Generators in keras for this problem.

def Generator(File_address, Batch_Size):
    while True:
      pickle_data = []
      with (open("myfile", "rb")) as openfile: #Read pickle file. this is a sample. you can use your way to read the pickle file.
          while True:
              pickle_data.append(pickle.load(File_address))
      
      for B in range(0, len(pickle_data), Batch_Size):
          X = pickle_data[B:B+Batch_Size]
          Y = Labels[B:B+Batch_Size] #Define your labels
          yield X, Y #Returning data for training.

now you can use your Generator:

train_gen = Generator('Address_to_pickle_file', Batch_Size)

Model.fit(train_gen, epochs=epoch, steps_per_epoch=Number_of_sampels//Batch_Size)

Hope this can be useful. good luck.

user_101440790
Related