tensorflow dataset splitting by participant

user_8615419 0
Viewed 81

I want to split a tf.data.DataSet on attributes, in my case participant or gesture. Currently, the dataset is a work in progress and the number of participants/gestures may grow. I originally set up a tfds config for this gestures-dataset, but I haven't figured out how to configure participant/gesture splitting here either.

How should I split the tf.data.DataSet object? Currently, my data set exists as a single tf_record. I would prefer to keep it that way instead of generating different files for each participant and gesture and have to regenerate all the gesture tf-records when a new participant is added.

this works (method 1, gross):

subset = ds.filter(lambda x: (x['participant'] == 1 or x['participant'] == 2))

this doesn't (method 2, dream):

subset = ds.filter(lambda x: any(x['participant'] == p for p in [1,2]))

OperatorNotAllowedInGraphError: using a tf.Tensor as a Python bool is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

I also attempted the same op as a decorated @tf.function.

example code with publicly available mnist dataset: juptyer notebook on colab

Is their a way to perform this operation in a similar style to method 2?

tensorflow python
1 Answers

You can solve it by:

def predicate(label, labels_to_filter):
  return tf.math.reduce_any(tf.equal(label, labels_to_filter))


labels_to_filter = tf.constant([0, 1, 2, 3, 4], dtype=tf.int64)
subset = dataset.filter(lambda x: predicate(x["participant"], labels_to_filter))
How it works:

tf.equal returns a boolean Tensor which contains True value if label matches one of the labels from labels_to_filter. tf.math.reduce_any returns True if there are True values in its input boolean Tensor.

user_141860870
Related