add LSTM/GRU to BERT embeddings in keras tensorflow

user_4304541 0
Viewed 2547

I am experimenting with BERT embeddings following this code https://github.com/strongio/keras-bert/blob/master/keras-bert.py

These are the important bits of the code (lines 265-267):

bert_output = BertLayer(n_fine_tune_layers=3)(bert_inputs)
dense = tf.keras.layers.Dense(256, activation="relu")(bert_output)
pred = tf.keras.layers.Dense(1, activation="sigmoid")(dense)

I want to add a GRU between BertLayer and the Dense layer

bert_output = BertLayer(n_fine_tune_layers=3)(bert_inputs)
gru_out = tf.keras.layers.GRU(100, activation='sigmoid')(bert_output)
dense = tf.keras.layers.Dense(256, activation="relu")(gru_out)
pred = tf.keras.layers.Dense(1, activation="sigmoid")(dense)

but I get this error TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'.

I am not entirely sure how to address this problem. Do I need to reshape bert_output or do I need to create an Embedding layer that the GRU can handle?

lstm tf.keras word-embedding
1 Answers

I have had the same error, the solution to this is

embedding_size = 768

bert_output = BertLayer(n_fine_tune_layers=3)(bert_inputs)
# Reshape bert_output before passing it the GRU
bert_output_ = tf.keras.layers.Reshape((max_seq_length, embedding_size))(bert_output)

gru_out = tf.keras.layers.GRU(100, activation='sigmoid')(bert_output_)
dense = tf.keras.layers.Dense(256, activation="relu")(gru_out)
pred = tf.keras.layers.Dense(1, activation="sigmoid")(dense)

I hope it works, you can refer to my question if needed

user_82795930
Related