I am trying to implement deep dreaming for a sound processing neural network and keep running into issues related to the handling of symbolic tensors, which I cannot seem to circumvent.
I narrowed the issue down to the fact that when functions are decorated with @tf.function, tensors passed to them are handled as symbolic tensors, which do not seem to share the same attributes as "regular" tensors and cannot be casted into other classes.
Below is a minimum reproducible example of the aforementioned issue:
import numpy as np
import tensorflow as tf
print(np.__version__)
print(tf.__version__)
tf.compat.v1.enable_eager_execution()
x = np.random.rand(35000,1)
x_tensor = tf.convert_to_tensor(x)
@tf.function
def some_function(some_tensor):
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
temp = some_tensor.numpy() # Error 2
temp = np.array(some_tensor) # Error 3
return(temp)
batched_x = tf.keras.preprocessing.timeseries_dataset_from_array(data=x_tensor, targets=None, sequence_length=256, sequence_stride=64)
temp = x_tensor.numpy()
temp = np.array(x_tensor)
temp = some_function(x_tensor)
Executing the code above throws the following error (Error 1):
1.19.5
2.4.1
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-61d46064abc7> in <module>()
22 temp = x_tensor.numpy()
23 temp = np.array(x_tensor)
---> 24 temp = some_function(x_tensor)
8 frames
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
975 except Exception as e: # pylint:disable=broad-except
976 if hasattr(e, "ag_error_metadata"):
--> 977 raise e.ag_error_metadata.to_exception(e)
978 else:
979 raise
TypeError: in user code:
<ipython-input-11-61d46064abc7>:15 some_function *
batched_tensor = tf.keras.preprocessing.timeseries_dataset_from_array(data=some_tensor, targets=None, sequence_length=256, sequence_stride=64) # Error 1
/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/preprocessing/timeseries.py:141 timeseries_dataset_from_array **
if sampling_rate <= 0 or sampling_rate >= len(data):
/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/ops.py:860 __len__
"shape information.".format(self.name))
TypeError: len is not well defined for symbolic Tensors. (some_tensor:0) Please call `x.shape` rather than `len(x)` for shape information.
Furthermore, commenting out the code inside the function one by one yields following issues:
Error 2:
AttributeError: 'Tensor' object has no attribute 'numpy'
Error 3:
NotImplementedError: Cannot convert a symbolic Tensor (some_tensor:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported
As you can see, the operations outside the function are perfectly executable but fail when performed on the symbolic tensor inside the function. I have seen this issue (or similar issues) being reported in different forums before but nowhere a satisfying solution was provided.
To compute the gradient ascend, I am dependent on batching the data using the tf.keras.preprocessing.timeseries_dataset_from_array before feeding it to the network to retrieve the activations. I am not looking to modify the network's architecture nor is passing the data preprocessed to the function an option for me. I feel like there should be a straight-forward way to manipulate symbolic tensors with tensorflow's built-in functions. Also, tf.make_ndarray() does not work.
Any help with either debugging the code or guiding me towards helpful references will be highly appreciated.
Thanks in advance!