Unsupported data type for TPU: double, caused by output IteratorGetNext:0

Question

I'm building a simple deep learning model with tf.keras in the following structure:


def build_simple_model(timestep, dim1, show_model_structure=False):

    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.LSTM(units=64, input_shape=(timestep, dim1), activation="relu", return_sequences=True, dtype="float64"))
    model.add(tf.keras.layers.LSTM(units=64, input_shape=(timestep, dim1), activation="relu", return_sequences=True, dtype="float64"))
    model.add(tf.keras.layers.LSTM(units=64, input_shape=(timestep, dim1), activation="relu", dtype="float64"))
    model.add(tf.keras.layers.Dense(32, activation="relu",dtype="float64")) 
    model.add(tf.keras.layers.Dense(1,dtype="float64"))

    # adam = Adam(lr=0.01, beta_1=0.9, beta_2=0.999, amsgrad=False)
    model.compile(loss='mse', 
                  optimizer='adam', 
                  metrics=['mape'])

    if show_model_structure:
        model.summary()
        draw_model_graph(model)
    return model

And employed the TPU on Colab:

resolver = tf.distribute.cluster_resolver.TPUClusterResolver()
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.experimental.TPUStrategy(resolver)

Build and fit the model with new TPU APIs:

with strategy.scope():
    test_model = build_simple_model(23, x_t.shape[2])
    test_model.fit(x_t.astype('float64'), y_train.astype('float64'), epochs=100, batch_size=64)#, callbacks=[callback])

But, the weird error still existed and complained about wrong data type.

Full trace stack:

---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_call(self, fn, *args)
   1364     try:
-> 1365       return fn(*args)
   1366     except errors.OpError as e:

11 frames
/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata)
   1347       # Ensure any changes to the graph are reflected in the runtime.
-> 1348       self._extend_graph()
   1349       return self._call_tf_sessionrun(options, feed_dict, fetch_list,

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _extend_graph(self)
   1387     with self._graph._session_run_lock():  # pylint: disable=protected-access
-> 1388       tf_session.ExtendSession(self._session)
   1389 

InvalidArgumentError: Unsupported data type for TPU: double, caused by output IteratorGetNext:0

During handling of the above exception, another exception occurred:

InvalidArgumentError                      Traceback (most recent call last)
 in ()
      1 with strategy.scope():
      2     test_model = build_simple_model(23, x_t.shape[2])
----> 3     test_model.fit(x_t.astype('float64'), y_train.astype('float64'), epochs=100, batch_size=64)#, callbacks=[callback])

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    725         max_queue_size=max_queue_size,
    726         workers=workers,
--> 727         use_multiprocessing=use_multiprocessing)
    728 
    729   def evaluate(self,

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training_distributed.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, **kwargs)
    617         validation_split=validation_split,
    618         shuffle=shuffle,
--> 619         epochs=epochs)
    620     if not dist_utils.is_distributing_by_cloning(model):
    621       with model._distribution_strategy.scope():

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/training.py in _distribution_standardize_user_data(self, x, y, sample_weight, class_weight, batch_size, validation_split, shuffle, epochs, allow_partial_batch)
   2270         session = None
   2271       else:
-> 2272         session = K.get_session()
   2273 
   2274       first_x_value = nest.flatten(x)[0]

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/backend.py in get_session(op_input_list)
    484   if not _MANUAL_VAR_INIT:
    485     with session.graph.as_default():
--> 486       _initialize_variables(session)
    487   return session
    488 

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/backend.py in _initialize_variables(session)
    901     # marked as initialized.
    902     is_initialized = session.run(
--> 903         [variables_module.is_variable_initialized(v) for v in candidate_vars])
    904     uninitialized_vars = []
    905     for flag, v in zip(is_initialized, candidate_vars):

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    954     try:
    955       result = self._run(None, fetches, feed_dict, options_ptr,
--> 956                          run_metadata_ptr)
    957       if run_metadata:
    958         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
   1178     if final_fetches or final_targets or (handle and feed_dict_tensor):
   1179       results = self._do_run(handle, final_targets, final_fetches,
-> 1180                              feed_dict_tensor, options, run_metadata)
   1181     else:
   1182       results = []

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
   1357     if handle is None:
   1358       return self._do_call(_run_fn, feeds, fetches, targets, options,
-> 1359                            run_metadata)
   1360     else:
   1361       return self._do_call(_prun_fn, handle, feeds, fetches)

/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py in _do_call(self, fn, *args)
   1382                     '
session_config.graph_options.rewrite_options.'
   1383                     'disable_meta_optimizer = True')
-> 1384       raise type(e)(node_def, op, message)
   1385 
   1386   def _extend_graph(self):

InvalidArgumentError: Unsupported data type for TPU: double, caused by output IteratorGetNext:0

Does any one know what's going on here?
Thanks in advance.

user_12635712 · Answer

I had the same problem running a keras model with TPU.

I discovered that it's related to the dtype of the y label (2nd) parameter provided to .fit. The only way it works smoothly is with dtype int64. any float type receives the same exception.

In my case, although I planned to run the model with binary-crossentropy which required float labels, I could work around it using categorical-crossentropy with int labels.

In your case I see that your loss is mse, so not sure you can workaround by using int labels.

Another way that succeeded to workaround the exception, but got other exceptions down the road, was as follows:

define the y label vector is tf.Variable:

sess = K.get_session()

...

y = tf.Variable(my_baseline_y_vector, dtype='bfloat16') # any float dtype EXCEPT float64

then before calling .fit:

train_y = y.eval(session=sess)

to translate it to an array that can be provided to .fit

model.fit(train_x, train_y, ...)

As said this passed the discussed dtype exception, but received other exceptions down the road :(. I gave up and am using categorical with int labels. If you don't have such a choice then I'll be happy to hear how you proceeded with the other exceptions :).

user_4007412 · Answer

I might not be of any help but it would be great if we could solve this together I get the same error and do not know how to resolve it

I am trying to complete a tutorial on machine learning and this is where I get stuck I have had to update the Tensorflow version from the tutorial as well as Keras version I think that maybe there could be some incompatibility issues

Unsupported data type for TPU: double, caused by output IteratorGetNext:0

2 Answers