Simulate streaming learning using Tensorflow's fit() and evaluate() built-in methods

user_12869054 0
Viewed 137

What I'm trying to achieve is to simulate a streaming learning method using Tensorflow's fit() and evaluate() methods.

What I have until now is a script like this, after getting some help from the community here:

import pandas as pd
import tensorflow as tf

df = pd.read_csv('labeled_tweets_processed.csv')

labels = df.pop('class')
dataset = tf.data.Dataset.from_tensor_slices((df, labels))

VOCAB_SIZE = 1000
encoder = tf.keras.layers.TextVectorization(
    max_tokens=VOCAB_SIZE)
encoder.adapt(dataset.map(lambda text, label: text))

BUFFER_SIZE = 2
BATCH_SIZE = 1

train_dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)

model = tf.keras.Sequential([
    encoder,
    tf.keras.layers.Embedding(
        input_dim=len(encoder.get_vocabulary()),
        output_dim=64,
        # Use masking to handle the variable sequence lengths
        mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
])

model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),
              optimizer=tf.keras.optimizers.Adam(1e-4),
              metrics=['accuracy'])

to setup the model and training the model using this command:

history = model.fit(train_dataset, epochs=1)

What I actually want to do is to simulate a Streaming environment where I have a pipeline like Predict -> Fit into the model.

I thought it could be accomplished by using a method like:

for x, y in enumerate(train_dataset):
  test_loss, test_acc = model.evaluate([x, y])
  model.fit(y)

but it doesn't seems to work right like this.

What is the right way to simulate the described environment?

What is the best way to iterate through dataset's each entry and input to the desired methods?

Thank you very much in advance!

Update 1:

What I have right now, but resulting in very low model accuracy. Not sure if the metrics are updated the right way.

for idx, (x, y) in enumerate(train_dataset):
  pred = model.predict_on_batch(x)
  print(model.test_on_batch(x, pred, reset_metrics=False, return_dict=True))
  model.train_on_batch(x, y, reset_metrics=False)
  print(f"After {idx} entries")
pandas keras tensorflow python
1 Answers

You can try something like this:

for idx, (x, y) in enumerate(train_dataset):
  test_loss, test_acc = model.evaluate(x, y)
  model.fit(x, y, epochs=1)

Update 1: Maybe try using a custom training loop:

import pandas as pd
import tensorflow as tf

df = pd.DataFrame(data = {'texts': ['Some text ssss', 'Some text', 'Some text', 'Some text', 'Some text'],
                          'class': [0, 0, 1, 1, 1]})

labels = df.pop('class')
dataset = tf.data.Dataset.from_tensor_slices((df, labels))

VOCAB_SIZE = 1000
encoder = tf.keras.layers.TextVectorization(
    max_tokens=VOCAB_SIZE)
encoder.adapt(dataset.map(lambda text, label: text))

BUFFER_SIZE = 2
BATCH_SIZE = 3

train_dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)

model = tf.keras.Sequential([
    encoder,
    tf.keras.layers.Embedding(
        input_dim=len(encoder.get_vocabulary()),
        output_dim=64,
        # Use masking to handle the variable sequence lengths
        mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64)),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1, activation='sigmoid')
])

opt = tf.keras.optimizers.Adam(1e-4)
loss_fn = tf.keras.losses.BinaryCrossentropy()
train_acc_metric = tf.keras.metrics.BinaryAccuracy()
test_acc_metric = tf.keras.metrics.BinaryAccuracy()
epochs = 2
for epoch in range(epochs):
    print("\nStart of epoch %d" % (epoch + 1,))
    
    for step, (x_batch_train, y_batch_train) in enumerate(train_dataset):
      pred = model(x_batch_train)
      test_acc_metric.update_state(y_batch_train, pred)
      print("Current test acc: %.4f" % (float(test_acc_metric.result()),))
      with tf.GradientTape() as tape:
          logits = model(x_batch_train, training=True)
          loss_value = loss_fn(y_batch_train, logits)
      grads = tape.gradient(loss_value, model.trainable_weights)
      opt.apply_gradients(zip(grads, model.trainable_weights))

      train_acc_metric.update_state(y_batch_train, logits)
      print("Current train acc: %.4f" % (float(train_acc_metric.result()),))

    test_acc = test_acc_metric.result()
    print("Total test acc over epoch: %.4f" % (float(test_acc),))
    test_acc_metric.reset_states()

    train_acc = train_acc_metric.result()
    print("Total train acc over epoch: %.4f" % (float(train_acc),))
    train_acc_metric.reset_states()

Start of epoch 1
Current test acc: 0.6922
Current train acc: 0.6922
Current test acc: 0.6936
Current train acc: 0.6936
Current test acc: 0.6928
Current train acc: 0.6928
Current test acc: 0.6934
Current train acc: 0.6934
Current test acc: 0.6938
Current train acc: 0.6938
Total test acc over epoch: 0.6938
Total train acc over epoch: 0.6938

Start of epoch 2
Current test acc: 0.6914
Current train acc: 0.6914
Current test acc: 0.6914
Current train acc: 0.6914
Current test acc: 0.6926
Current train acc: 0.6926
Current test acc: 0.6932
Current train acc: 0.6932
Current test acc: 0.6936
Current train acc: 0.6936
Total test acc over epoch: 0.6936
Total train acc over epoch: 0.6936
user_96578610
Related