How to manipulate symbolic tensors in TensorFlow

Viewed 1074

I am trying to implement deep dreaming for a sound processing neural network and keep running into issues related to the handling of symbolic tensors, which I cannot seem to circumvent.

I narrowed the issue down to the fact that when functions are decorated with @tf.function, tensors passed to them are handled as symbolic tensors, which do not seem to share the same attributes as "regular" tensors and cannot be casted into other classes.

Below is a minimum reproducible example of the aforementioned issue:

import numpy as np
import tensorflow as tf

print(np.__version__)
print(tf.__version__)

tf.compat.v1.enable_eager_execution()

x = np.random.rand(35000,1)

x_tensor = tf.convert_to_tensor(x)

@tf.function
def some_function(some_tensor):
  batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
  temp = some_tensor.numpy() # Error 2
  temp = np.array(some_tensor) # Error 3

  return(temp)

batched_x = tf.keras.preprocessing.timeseries_dataset_from_array(data=x_tensor, targets=None, sequence_length=256, sequence_stride=64)
temp = x_tensor.numpy()
temp = np.array(x_tensor)
temp = some_function(x_tensor)

Executing the code above throws the following error (Error 1):

1.19.5
2.4.1

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-11-61d46064abc7> in <module>()
     22 temp = x_tensor.numpy()
     23 temp = np.array(x_tensor)
---> 24 temp = some_function(x_tensor)

8 frames

/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    975           except Exception as e:  # pylint:disable=broad-except
    976             if hasattr(e, "ag_error_metadata"):
--> 977               raise e.ag_error_metadata.to_exception(e)
    978             else:
    979               raise

TypeError: in user code:

    <ipython-input-11-61d46064abc7>:15 some_function  *
        batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/preprocessing/timeseries.py:141 timeseries_dataset_from_array  **
        if sampling_rate <= 0 or sampling_rate >= len(data):
    /usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:860 __len__
        "shape information.".format(self.name))

    TypeError: len is not well defined for symbolic Tensors. (some_tensor:0) Please call `x.shape` rather than `len(x)` for shape information.

Furthermore, commenting out the code inside the function one by one yields following issues:

Error 2:

AttributeError: 'Tensor' object has no attribute 'numpy'

Error 3:

NotImplementedError: Cannot convert a symbolic Tensor (some_tensor:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

As you can see, the operations outside the function are perfectly executable but fail when performed on the symbolic tensor inside the function. I have seen this issue (or similar issues) being reported in different forums before but nowhere a satisfying solution was provided.

To compute the gradient ascend, I am dependent on batching the data using the tf.keras.preprocessing.timeseries_dataset_from_array before feeding it to the network to retrieve the activations. I am not looking to modify the network's architecture nor is passing the data preprocessed to the function an option for me. I feel like there should be a straight-forward way to manipulate symbolic tensors with tensorflow's built-in functions. Also, tf.make_ndarray() does not work.

Any help with either debugging the code or guiding me towards helpful references will be highly appreciated.

Thanks in advance!

1 Answers

I recommend you take a look at some articles that explain eager vs graph execution mode such as:

https://towardsdatascience.com/eager-execution-vs-graph-execution-which-is-better-38162ea4dbf6

Tensorflow has eager tensors which can be converted to numpy values and symbolic tensors which represent nodes in an execution graph.

When you decorate a function with @tf.function you are marking this function as a graph function. tensorflow will execute this function in python once in order to build a graph; this graph is then executed each time the model node is called; this execution is done via the tensorflow core library and can happen on CPU, GPU, etc... As such it is not meaningful to call .numpy() on a symbolic tensor as the graph executor does not have access to a python runtime.

You can however build models that mix eager and graph nodes... be aware that any eager nodes will be rather slow.

Eager tensors have their place when one is attempting to debug the math of a node; but in general, since execution speed is important, it is more useful to think in terms of graphs and symbolic tensors.

Related