How can I replace LSTM with BERT?

user_16551510 0
Viewed 137

I need to use BERT instead of LSTM in this example. Is it possible to just replace the word LSTM with BERT?

inputs1=Input(shape=(2048,))
  
fe1 = Dense(128, activation='relu')(inputs1)

inputs2 = Input(shape=(max_length,))

se1 = Embedding(vocab_size, 256, mask_zero=True)(inputs2)

se2 = LSTM(256)(se1)

se3 = Dropout(0.5)(se2)

decoder1 = Concatenate()([fe1, se3])

decoder2 = Dense(128, activation='relu')(decoder1)

outputs = Dense(vocab_size, activation='softmax')(decoder2)

model=Model(inputs=[inputs1,inputs2],outputs=outputs)

or how can I start to use it, please?

I tried this:

inputs1=Input(shape=(2048,))
fe1 = Dense(128, activation='relu')(inputs1)
inputs2 = Input(shape=(max_length,), name="input_ids")

in_mask = Input(shape=(max_length,), name="input_masks")
in_segment = Input(shape=(max_length,), name="segment_ids")

bert_inputs = [inputs2, in_mask, in_segment] 
bert_output = BertLayer(n_fine_tune_layers=12, pooling="mean")(bert_inputs) 

decoder1 = Concatenate()([fe1 bert_output])

decoder2 = Dense(256, activation='relu')(decoder1)

outputs = Dense(vocab_size, activation='softmax')(decoder2)
    
model=Model(inputs=[inputs1,inputs2],outputs=outputs)

but got:

 bert_output = BertLayer(n_fine_tune_layers=12, pooling="mean")(bert_inputs) 
TypeError: __init__() missing 2 required positional arguments: 'pretrained_model_path' and 'output_size'
bert-language-model lstm
2 Answers

BERT is a pretraining technique. It uses transfer learning to use previous knowledge in a new setting. Thus, to use BERT you need to specify a pre-trained model name or path. So, NO, you cannot simply replace LSTM to BERT.

Look at the answers to this question: How to create a BERT Layer with Keras?

user_24199600

I am trying to use Sentiment Classification with LSTM and pre-trained BERT embeddings, and later language translation with Transformer first of all I downloaded

!pip install ktrain
!pip install tensorflow_text

And I imported the necessary lib

import pathlib
import random
import numpy as np
from typing import Tuple, List

import matplotlib.pyplot as plt
import matplotlib.ticker as ticker
from sklearn.model_selection import train_test_split

# tensoflow imports
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras.layers import (
    TextVectorization, LSTM, Dense, Embedding, Dropout,
    Layer, Input, MultiHeadAttention, LayerNormalization)
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.initializers import Constant
from tensorflow.keras import backend as K
import tensorflow_text as tf_text 
import ktrain
from ktrain import text

And I downloaded and extracted Large Movie dataset from Stanford

!wget https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz
!tar -xzf aclImdb_v1.tar.gz

1- I try to use LSTM with train by Creating the training and test sets with the texts_from_folder function of the ktrain.text module

%reload_ext autoreload
%autoreload 2
%matplotlib inline
import os
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID";
os.environ["CUDA_VISIBLE_DEVICES"]="0";

DATADIR ='/content/aclImdb'
trn, val, preproc = text.texts_from_folder(DATADIR,max_features=20000, maxlen=400, ngram_range=1,                                               preprocess_mode='standard', train_test_names=['train', 'test'],classes=['pos', 'neg'])

And I am trying to build LSTM model her

K.clear_session()    
def build_LSTM_model(
        embedding_size: int,
        total_words: int,
        lstm_hidden_size: int,
        dropout_rate: float) -> Sequential:
    model.add(Embedding(input_dim = total_words,output_dim=embedding_size,input_length=total_words))
    model.add(LSTM(lstm_hidden_size,return_sequences=True,name="lstm_layer"))
    model.add(GlobalMaxPool1D())
#     model.add(Dense(total_words, activation='softmax'))
    model.add(Dropout(dropout_rate))
    model.add(Dense(MAX_SEQUENCE_LEN, activation="relu"))
#     adam = Adam(lr=0.01)
    model.compile(loss='CategoricalCrossentropy', optimizer=Adam(lr=0.01), metrics=['CategoricalAccuracy'])
    model.summary()
    model = Sequential()

with the following requirements for a sequential model The model should include: One Embedding layer at the beginning. (Watch out for proper parameterization!) At least one LSTM layer.At least one Dropout layer for regularization.One final Dense layer mapping to the outputs. compile model, with categorical_crossentropy loss and the adam optimizer. or might want to add other types of metrics for example CategoricalAccuracy makes sense here.

And then I want to use the ktrain library's get_learner method to create an easily trainable version of the previous model. and to use test set as the val_data, to see the performance. ( not include the proper train-validation-test split, but it could be extended if required.) I am using the learner's lr_find and lr_plot methods to determine the most effective learning rate for the model. by Specifying the max_epochs parameter of lr_find to limit the time this takes. a couple of epochs! to determine the best learning rate based on the plot. and find balance between the fastest convergence and stability

learner: ktrain.Learner
model = text.text_classifier('bert', trn , preproc=preproc)
learner.lr_find()
learner.lr_plot()
learner.fit_onecycle(1e-4, 1)

I faced following errors ValueError Traceback (most recent call last) in () 6 # workers=8, use_multiprocessing=False, batch_size=64) 7 ----> 8 model = text.text_classifier('bert', trn , preproc=preproc)

     10 # learner.lr_find()
1 frames
/usr/local/lib/python3.7/dist-packages/ktrain/text/models.py in _text_model(name, train_data, preproc, multilabel, classification, metrics, verbose)
    109         raise ValueError(
    110             "if '%s' is selected model, then preprocess_mode='%s' should be used and vice versa"
--> 111             % (BERT, BERT)
    112         )    113     is_huggingface = U.is_huggingface(data=train_data)
ValueError: if 'bert' is selected model, then preprocess_mode='bert' should be used and vice versa

And next step to make it with LSTM with pretrained static word vectors

user_147298200
Related