Tensorflow - LSTM, TextVectorization (custom standardization) and ModelCheckpoint

Question

I'm fairly new to Tensorflow. I've been following Word Embeddings, Basic text classification and RNN articles on TF website.

I have dataset_dir = 'aclImdb' IMDB dataset as in those tutorials and everything is the same as described there. I've added a ModelCheckpoint with save_best_only=True and I'm getting two errors - first when training, between epochs, I guess when the best model is saved and second one when the best saved model is loaded.

Warning 1: WARNING:absl:Found untraced functions such as lstm_cell_1_layer_call_fn, lstm_cell_1_layer_call_and_return_conditional_losses, lstm_cell_2_layer_call_fn,lstm_cell_2_layer_call_and_return_conditional_losses, lstm_cell_4_layer_call_fn while saving (showing 5 of 20). These functions will not be directly callable after loading.

WARNING:absl: has the same name 'LSTMCell' as a built-in Keras object. Consider renaming to avoid naming conflicts when loading with 'tf.keras.models.load_model'. If renaming is not possible, pass the object in the 'custom_objects' parameter of the load function.

Error 2: RuntimeError: Unable to restore a layer of class TextVectorization. Layers of class TextVectorization require that the class be provided to the model loading code, either by registering the class using '@keras.utils.register_keras_serializable' on the class def and including that file in your program, or by passing the class in a 'keras.utils.CustomObjectScope' that wraps this load call.

My variables:

batch_size = 64
seed = 42
max_features = 10000  # vocabulary size
sequence_length = 250  # sentence length of reviews or tweets, etc.
output_dim = 16  # encoded size of a vector

Here's my callback:

def create_model_checkpoint(model_name, save_path='model_experiments'):
  return tf.keras.callbacks.ModelCheckpoint(filepath=os.path.join(save_path, model_name),
                                            verbose=0, # output limited amount of info
                                            save_best_only=True)

Here's TextVectorization (the same as in the articles):

def custom_standardization(input_data):
    lowercase = tf.strings.lower(input_data)
    stripped_html = tf.strings.regex_replace(lowercase, '
', ' ')
    return tf.strings.regex_replace(stripped_html,
                                    '[%s]' % re.escape(string.punctuation), '')

vectorize_layer = tf.keras.layers.TextVectorization(
    standardize=custom_standardization,
    max_tokens=max_features,
    output_mode='int',
    output_sequence_length=sequence_length
)

Here's the Model (I have added GlobalAveragePooling1D and callback compared to the article):

model_2 = tf.keras.models.Sequential([
    vectorize_layer,
    tf.keras.layers.Embedding(len(vectorize_layer.get_vocabulary()), 64, mask_zero=False),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(32, return_sequences=True)),
    tf.keras.layers.GlobalAveragePooling1D(),
    tf.keras.layers.Dense(64, activation='relu'),
    tf.keras.layers.Dense(1)
], name='model_2_Bidirectional')

model_2.compile(loss=tf.keras.losses.BinaryCrossentropy(),
                optimizer=tf.keras.optimizers.Adam(),
                metrics=['accuracy'])

history_model_2 = model_2.fit(train_ds, epochs=3,
                              validation_data=valid_ds,
                              callbacks=[create_model_checkpoint(model_name=model_2.name)])

model_2_loaded = tf.keras.models.load_model(os.path.join('model_experiments', model_2.name))
model_2_loaded.evaluate(valid_ds)

Questions: Regarding Warning - I have tried defining name='lstm_custom_64' and 'lstm_custom_32' for the layers but these warnings still kept on coming up. I couldn't find any clear solution to what this warning means and how to fix it.

Regarding Error - I have more clues on this one, I guess I have to define @keras.utils.register_keras_serializable and write custom layer class inheriting from TextVectorization and include custom_standardization into it. Should I? I would love if you could share the solution for this one as I'm not experienced with TF.

P.S. What about adapt? as far as I understand this creates a vocab. Is this getting saved and then loaded when performing save/load?

text_ds = train_ds.map(lambda x, y: x)
vectorize_layer.adapt(text_ds)

Tensorflow - LSTM, TextVectorization (custom standardization) and ModelCheckpoint

0 Answers