efficiently converting for model.fit

Question

I'm struggling in loading data into model.fit efficiently. My code creates training_data object with samples and values. Samples is a standard python list of objects of tf.Tensor class. Values is a list of integers.

When running

model.fit(training_data.samples, training_data.values, epochs=10)

I get an error

ValueError: Failed to find data adapter that can handle input: ( containing values of types {""}), ( containing values of types {""})

I can get this two work, by pre-converting it all to numpy arrays like this:

s, v = np.asarray(training_data.samples), np.asarray(training_data.values)
model.fit(s, v, epochs=10)

However this is impossibly slow. Loading data and very heavy preprocessing (signal chunking, fft, etc. etc.) takes about a minute and then just the data conversion with 1800 samples this part hangs for an hour and I lose patience before actual learning starts. Tensors's shape is (94, 257) so nothing big.

So what's an efficient way to pass data to model.fit, given that I already have it in memory.

user_7617767 · Answer

Hi this is just a suggestion but try using a generator object from tf.keras.utils.Sequence but this also depends on what type of data you are using?

You can look at the example here: https://stanford.edu/~shervine/blog/keras-how-to-generate-data-on-the-fly

I implemented my own here you can check it out: https://github.com/edwin-19/custom_keras_generator/blob/master/notebooks/Model%20Comparison.ipynb

user_10016052 · Answer

For me it's a common error when I use the wrong input format (typically when I pass a list).

Try to convert only training_data.values to a numpy array or a tensor if it's a list.

As pinpointed in the documentation (x, y, validation_data,..) have a limited number of accepted inputs, i.e. numpy array and tensor: model.fit

efficiently converting for model.fit

2 Answers