I've a generator function that yields the following tuple: yield (transformed_input_array, set_y)
transformed_input_array is a list of ndarrays with the following shape: (1024, 104), (1024, 142), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 140) and the following types: tf.float64, tf.float64, tf.int8, tf.int16, tf.int8, tf.int8, tf.float64 set_y is a ndarray of shape 1024 and type of int64
I've wrapped my generator with tf.data.Dataset.from_generator function, here is the code:
dataset = tf.data.Dataset.from_generator(
generator,
# output_signature=(
# tf.TensorSpec(shape=(), dtype=(tf.float64, tf.float64, tf.int8, tf.int16, tf.int8, tf.int8, tf.float64)),
# tf.TensorSpec(shape=1024, dtype=tf.int64))
output_types=(tf.float64, tf.float64, tf.int8, tf.int16, tf.int8, tf.int8, tf.float64, tf.int64),
output_shapes=((1024, 104), (1024, 142), (1024, 1), (1024, 1), (1024, 1), (1024, 1), (1024, 140), 1024)
)
But when I run the training, I get the following error:
ValueError: Data is expected to be in format
x,(x,),(x, y), or(x, y, sample_weight), found: (<tf.Tensor 'IteratorGetNext:0' shape=(1024, 104) dtype=float64>, <tf.Tensor 'IteratorGetNext:1' shape=(1024, 142) dtype=float64>, <tf.Tensor 'IteratorGetNext:2' shape=(1024, 1) dtype=int8>, <tf.Tensor 'It eratorGetNext:3' shape=(1024, 1) dtype=int16>, <tf.Tensor 'IteratorGetNext:4' shape=(1024, 1) dtype=int8>, <tf.Tensor 'IteratorGetNext:5' shape=(1024, 1) dtype=int8>, <tf.Tensor 'IteratorGetNext:6' shape=(1024, 140) dtype=float64>, <tf.Tensor 'ExpandDims:0' shape=(1024, 1) dtype=int64>)
If I try to run with output_signature param (commented out code), I get the following error:
TypeError: Cannot convert value (tf.float64, tf.float64, tf.int8, tf.int16, tf.int8, tf.int8, tf.float64) to a TensorFlow DType.
Can someone provide an example, of how I should treat complex type (list of ndarrays)? Couldn't find any example in TF documentation..