Python Membership Operators "In" TensorFlow Datasets

user_6586216 0
Viewed 249

I'm developing an input pipeline with TensorFlow Datasets, the dataset has just two columns, I want to filter based in a list of values, but I just can filter the dataset using the operator equal "==", when I tried to use the membership operator "in" I received the following error.

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.

Below my code:

import numpy as np
import tensorflow as tf


# Load file
file_path = 'drive/My Drive/Datasets/category_catalog.csv.gz'

def get_dataset(file_path, batch_size=5, num_epochs=1, **kwargs):
    return tf.data.experimental.make_csv_dataset(
        file_path,
        batch_size=batch_size,
        na_value="?",
        num_epochs=num_epochs,
        ignore_errors=True,
        **kwargs
    )

raw_data = get_dataset(
    file_path,
    select_columns=['description', 'department'],
    compression_type='GZIP'
)

This filter works:

@tf.function
def filter_fn(features):
    return features['department'] == 'MOVEIS'

ds = raw_data.unbatch()
ds = ds.filter(filter_fn)
ds = ds.batch(2)

Output:

next(iter(ds))

OrderedDict([('description', <tf.Tensor: shape=(2,), dtype=string, numpy=
              array([b'KIT DE COZINHA KITS PARANA 8 PORTAS GOLDEN EM MDP LINHO BRANCO E LINHO PRETO',
                     b'ARMARIO AEREO PARA COZINHA 1 PORTA HORIZONTAL EXCLUSIVE ITATIAIA PRETO MATTE'],
                    dtype=object)>),
             ('department',
              <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'MOVEIS', b'MOVEIS'], dtype=object)>)])

This filter doesn't work:

@tf.function
def filter_fn(features):
    return features['department'] in ['FERRAMENTAS', 'MERCEARIA', 'MOVEIS']

ds = raw_data.unbatch()
ds = ds.filter(filter_fn)
ds = ds.batch(2)

Error:

---------------------------------------------------------------------------
OperatorNotAllowedInGraphError            Traceback (most recent call last)
<ipython-input-52-52131b5369b6> in <module>()
      6 
      7 ds = raw_data.unbatch()
----> 8 ds = ds.filter(filter_fn)
      9 ds = ds.batch(2)

18 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    966           except Exception as e:  # pylint:disable=broad-except
    967             if hasattr(e, "ag_error_metadata"):
--> 968               raise e.ag_error_metadata.to_exception(e)
    969             else:
    970               raise

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-52-52131b5369b6>:5 filter_fn  *
        return features['department'] in ['FERRAMENTAS', 'MERCEARIA', 'MOVEIS']
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:778 __bool__
        self._disallow_bool_casting()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:545 _disallow_bool_casting
        "using a `tf.Tensor` as a Python `bool`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:532 _disallow_when_autograph_enabled
        " decorating it directly with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: using a `tf.Tensor` as a Python `bool` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.

Here the link to Colab where it possible to run and check the error:

tensorflow-datasets tensorflow python
2 Answers

What I tried is the following for the filter function:

tf.reduce_any(tf.math.equal(features['department'], ['FERRAMENTAS', 'MERCEARIA', 'MOVEIS']))

It seems to work in the colab you provided but I am not sure it's what you want.

The logic is the following: the math.equal operator will give a tensor of size 3, where each entry is True or False. The first entry is whether the department is "FERRAMENTAS", etc... Then the reduce_any will basically perform a logical OR on this 3 entry tensor. So if the department is one of the 3 white listed ones, it will have exactly one True entry in the 3 entry tensor and therefore the reduce_any output will be True. It will be False in all other cases.

user_43325850

Even though the previous answer solved your problem, I want to propose a more general answer for the persons who will come here in the future.

Suppose that you have an arbitrary tensor x, such as:

>>> x = tf.range(20)
>>> x
<tf.Tensor: shape=(20,), dtype=int32, numpy=
array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
       17, 18, 19])>

If we want to get the positions of some elements, like 4, 11, 14, we can store them in a tensor y:

>>> y = tf.constant([4, 11, 14])
>>> y
<tf.Tensor: shape=(3,), dtype=int32, numpy=array([ 4, 11, 14])>

Then use an equal operation between the values vector x and the transpose of the searched elements vector y. The result will be an array of booleans with 2 dimensions (length of x, length of y). This array should be reduced into a vector with the same length of x by using tf.reduce_any on axis 0:

>>> tf.reduce_any(x == tf.reshape(y, (-1, 1)), axis=0)
<tf.Tensor: shape=(20,), dtype=bool, numpy=
array([False, False, False, False,  True, False, False, False, False,
       False, False,  True, False, False,  True, False, False, False,
       False, False])>

The positions having a True value are the ones where the elements of y are located inside x.

Now if you want just to perform a membership test of wether any element of x in in y, you can just drop the axis=0 argument:

>>> tf.reduce_any(x == tf.reshape(y, (-1, 1)))
<tf.Tensor: shape=(), dtype=bool, numpy=True>

This solution can be generalized to higher dimensions of x and y by changing the second parameter of tf.reshape to add another dimension to y and following the same logic.

user_50089680
Related