DataGenerator generates indexes out of bounds

Viewed 31

I trying to fine tune BERT model. For this purpose i use DataGeneartor

class BertSemanticDataGenerator(tf.keras.utils.Sequence):
    """Generates batches of data."""

def __init__(
        self,
        sentence_pairs,
        labels,
        batch_size=batch_size,
        shuffle=True,
        include_targets=True,
):
    self.sentence_pairs = sentence_pairs
    self.labels = labels
    self.shuffle = shuffle
    self.batch_size = batch_size
    self.include_targets = include_targets
    # Load our BERT Tokenizer to encode the text.
    # We will use base-base-uncased pretrained model.
    self.tokenizer = transformers.BertTokenizer.from_pretrained("bert-base-uncased", do_lower_case=True)
    self.indexes = np.arange(len(self.sentence_pairs))
    self.on_epoch_end()

def __len__(self):
    # Denotes the number of batches per epoch.
    return len(self.sentence_pairs) // self.batch_size

def __getitem__(self, idx):
    # Retrieves the batch of index.
    indexes = self.indexes[idx * self.batch_size: (idx + 1) * self.batch_size]
    sentence_pairs = self.sentence_pairs[indexes]

    # With BERT tokenizer's batch_encode_plus batch of both the sentences are
    # encoded together and separated by [SEP] token.
    encoded = self.tokenizer.batch_encode_plus(
        sentence_pairs.tolist(),
        add_special_tokens=True,
        max_length=max_length,
        return_attention_mask=True,
        return_token_type_ids=True,
        padding=True,
        return_tensors="tf",
    )

    # Convert batch of encoded features to numpy array.
    input_ids = np.array(encoded["input_ids"], dtype="int32")
    attention_masks = np.array(encoded["attention_mask"], dtype="int32")
    token_type_ids = np.array(encoded["token_type_ids"], dtype="int32")

    # Set to true if data generator is used for training/validation.
    if self.include_targets:
        labels = np.array(self.labels[indexes], dtype="int32")
        return [input_ids, attention_masks, token_type_ids], labels
    else:
        return [input_ids, attention_masks, token_type_ids]

The size of train is 27051 / test is 9017, the batch size is 32. When i trying to generates labels labels = np.array(self.labels[indexes], dtype="int32") the generator returns labels that out of bounds and i got an error

KeyError: '[7326, 2726, 23864, 4084, 3394, 19934, 22494] not in index'

There is probably something wrong in indexes calculation indexes = self.indexes[idx * self.batch_size: (idx + 1) * self.batch_size] but i can't get it

1 Answers

When i was splitting dataframe to train / val / test

X_train, X_val, y_train, y_val = train_test_split(X, y, stratify=y)
X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size=0.05, random_state=1)

i forgot to drop indexes. This fixed the issue

for df in (X_train, X_val, X_test, y_train, y_val, y_test):
    df.reset_index(drop=True, inplace=True)
Related