Optimizing a Tensorflow Data Pipeline With Mixed Datatypes

user_7189173 0
Viewed 163

Edit:

Since I was asked for code. Here's my model:

def def_model():
  image_input = Input(shape=(200,200,3), name="image") 
  feature_input = Input(shape=(500,5), name="time")
  output = decoder(feature_input,image_input,10,10)
  model = Model(
      inputs=[image_input, feature_input],
      outputs=output,
    )
  opt_adam = tf.keras.optimizers.Adam(clipnorm=0.5, epsilon=0.1)
  model.compile(loss=tf.losses.MeanSquaredError(), optimizer=opt_adam)
  return(model)

My current training loops contain a version of the following data inputs:

img_batch = tf.stack(image_list,axis=0)
inputs_batch = tf.reshape(tf.stack([inputs]*8,axis=0),[8,500,5])
outputs_batch = tf.reshape(tf.stack([outputs]*8,axis=0),[8,500,1])
model.fit({"image": img_batch, "time_features": inputs_batch},
          outputs_batch,
          epochs=3,
          batch_size = 1,
          callbacks=[tensorboard_callback]
          )

I checked using Tensorboard, and the loop itself is probably the least efficient part of the training (the gap between each call to fit is larger than the amount of time it spends training).

Original Question

I am training a model with mixed datatypes, and it is quite slowly. So, I am trying to speed it up. Usually, I would attempt to do this by turning my data into one tensor. However, that is probably not an option this time around. I have two inputs, a time series and an image, and I have a time series output.

The time series' have shape (length_of_sequence,5) and (length_of_sequence,1), and both are floats. The image is a square (n,n) image with integer values. I guess in principle I could turn these into two separate tensors, but what are my best options in creating a data pipeline?

tensorflow python
2 Answers
user_67709160

If your ramp-up for a fit is long, this points to a memory I/O issue.

(the gap between each call to fit is larger than the amount of time it spends training)

From a performance perspective, it seems weird that you access 2 different variables alternatingly in a loop. As a "preconditioning" step, can you combine the data from img_batch and inputs_batch into 1 auxiliary variable (with adjusted shape) before fitting? How does your runtime performance look like when only accessing this 1 variable in your loop?

user_147754780
Related