Handling text data with Tensorflow 2 "ERROR: Attempted to pad to a smaller size than the input element"

user_10908375 0
Viewed 709

I'm getting this error while iterating through a tokenized text dataset:

tensorflow.python.framework.errors_impl.DataLossError: Attempted to pad to a smaller size than the input element

It probably stems from the fact that I simply cannot understand the padded_shapes=([None], (1,)) argument. Here's the dataset and the code:

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
import tensorflow as tf
import tensorflow_datasets as tfds
from collections import Counter

os.chdir('/home/nicolas/Documents/Datasets')

data = tf.data.experimental.CsvDataset('rotten_tomatoes_string.csv',
                                       record_defaults=[tf.string, tf.string],
                                       header=True, select_cols=[0, 1])  

review_tokenizer= tfds.features.text.Tokenizer()
vocabulary = Counter()

for ix, (_, review) in enumerate(data):
    review = tf.strings.lower(review)
    tokens = review_tokenizer.tokenize(review.numpy())
    vocabulary.update(tokens)

vocabulary = [key for (key, value) in vocabulary.items() if value >= 5 and len(key) > 2]

ratings = {'rotten', 'fresh'}
review_encoder = tfds.features.text.TokenTextEncoder(vocabulary)
freshness_encoder = tfds.features.text.TokenTextEncoder(ratings)

def encode(freshness, review):
    review = review_encoder.encode(review.numpy())
    freshness = freshness_encoder.encode(freshness.numpy())
    return freshness, review

def tf_map(freshness, review):
    encoded_freshness, review = tf.py_function(encode,
                                               inp=[freshness, review],
                                               Tout=[tf.int32, tf.int32])
    review.set_shape([None])
    encoded_freshness.set_shape(1,)
    return encoded_freshness, review


TAKE = 480_000

train_data = data.map(tf_map).take(1000)
train_data = train_data.padded_batch(8, padded_shapes=([None], (1,)))

next(iter(train_data))

How can I deal with this?

tensorflow-datasets keras tensorflow nlp python
1 Answers

With padded_shapes=([None], (1,)) you are requesting the second element to be padded to size of 1. But, you should consider that:

  1. The second element is the review tokens (look at the return statement of tf_map function) and it probably consists of tens or hundreds of values (for each sample).

  2. The padded_batch method of Dataset does not do any truncating; it can only do padding.

Therefore, according to the two points above, you would get the mentioned error in your question. Instead, use a higher constant dimension size (which should be greater that the length of the all the samples in your dataset; otherwise you would get the same error), or use [None] to pad each batch to the smallest size that fits (i.e. to the maximum length of the samples in each batch).

user_20996070
Related