Is it possible to apply some transformation ( augmentation) of Text data Dynamically during training in Tensorflow?

user_11725056 0
Viewed 119

Let us suppose I have this very simple pipeline which already works:

X = [['she let the balloon float up into the air with her hopes and dreams'],
        ['the old rusted farm equipment surrounded the house predicting its demise'],
        ['he was so preoccupied with whether or not he could that he failed to stop to consider if he should']] # three samples

Y = [0,1,1] # Three samples

train_X, val_X, train_y, val_y = train_test_split(X,Y, test_size=0.1, stratify = Y, random_state = SEED) # Scikit learn's

vocab_size = 10 
tokenizer = Tokenizer(num_words=vocab_size, filters = ' ') # Keras Tokenizer
tokenizer.fit_on_texts(list(train_X))
train_X = tokenizer.texts_to_sequences(train_X)
val_X = tokenizer.texts_to_sequences(val_X)

## Pad the sentences 
train_X = pad_sequences(train_X, maxlen=maxlen)
val_X = pad_sequences(val_X, maxlen=maxlen)


history = model.fit(
    train_X, train_y, batch_size=8, epochs = 2, validation_split = 0.1) # compiled model

And I have a function (I already have some function which does the transformation):

def augment_text_data(string):
  temp = string.split(' ')
  temp[np.random.randint(0,len(temp)] = 'some_word'
  return ' '.join(temp)

Now if I want to apply this transformation of data during runtime, what will I have to do?

One option is that I transform my whole data first during training but I want to do something what ImageDataGenerator does for images where you can just pass the preprocessing_function.

Thing is that

  1. I am using Embedding (Model of Custom trained with Gensim)
  2. On top of that, I am experimenting with tf-idf weighted Embedding (But ignore that for now)
  3. We need to Tokenize texts first which makes the process I think, impossible

Can someone suggests me what is the right approach to do given my data and so?

tensorflow2.0 keras tensorflow deep-learning nlp
1 Answers

When I train my models I use the map function in the input pipeline of my dataset. Here is a guide: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#map

You need to define a function with the transformation you want to perform during training. This function must take a sample from your dataset as input (idealy a batch). Then you perform the augmentation and return the augmented sample. Your code could look s.th. like this:

#define function
def augmentationFunc(sample):
  augmentedSample = someTransformation(sample)
  return augmentedSample

#create dataset
dataset = tf.data.Dataset.from_tensor_slices(dataTensor)

#define inputpipeline
dataset = ( 
           dataset
           .shuffle(...)
           .batch(...)
           .map(augmentationFunc)
           )

Best, Sascha

user_179057640
Related