BERT Multi-class text classification in Google Colab

user_9994102 0
Viewed 6082

I'm working on a data set of social media comments(including youtube links) as input features and the Myers-Biggs Personality Profile as the target label: 

    type    posts
0   INFJ    'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   ENTP    'I'm finding the lack of me in these posts ver...
2   INTP    'Good one _____ https://www.youtube.com/wat...
3   INTJ    'Dear INTP, I enjoyed our conversation the o...
4   ENTJ    'You're fired.|||That's another silly misconce...

but from what I've found, BERT wants DataFrame's to be in this format:

a   label   posts
0   a   8   'http://www.youtube.com/watch?v=qsXHcwe3krw|||...
1   a   3   'I'm finding the lack of me in these posts ver...
2   a   11  'Good one _____ https://www.youtube.com/wat...
3   a   10  'Dear INTP, I enjoyed our conversation the o...
4   a   2   'You're fired.|||That's another silly misconce...

The resulting output must be a prediction on a test set of comments split into four columns, one for each Personality Profile where, for example, 'Mind' = 1 is the label for Extrovert. Basically splitting a type like INFJ into 'Mind','Energy','Nature','Tactics', like such:

    type    post    Mind    Energy  Nature  Tactics
0   INFJ    'url-web    0   1   0   1
1   INFJ    url-web 0   1   0   1
2   INFJ    enfp and intj moments url-web sportscenter n... 0   1   0   1
3   INFJ    What has been the most life-changing experienc...   0   1   0   1
4   INFJ    url-web url-web On repeat for most of today.    0   1   0   1

I've installed pytorch-pretrained-bert using:

!pip install pytorch-pretrained-bert

I've imported the models and tried to tokenize the 'posts' column using:

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

tokenized_train = tokenizer.tokenize(train)

but receive this error:

TypeError: ord() expected a character, but string of length 5 found

I tried this based off the pytorch-pretrained-bert GitHub Repo and a Youtube vidoe.

I am a Data Science intern with no Deep Learning experience at all. I simply want to experiment with the BERT model in the most simplest way to predict the multi-class classified output so I can compare the results to simpler text-classification models we are currently working on. I am working in Google Colab and the resulting output should be a .csv file.

I understand that this is a complicated model and all documentation and examples surrounding the model are complex (fine tuning layers etc.) but any help for a simple implementation(if in fact there is such a thing) for a beginner Data Scientist with minimal Software Engineering experience, would be much appreciated.

bert-language-model data-science pytorch google-colaboratory python
2 Answers
user_49033030

Simpler is a subjective term. Assuming you are open to use Tensorflow and keras-bert, you could do Multi-class text classification using BERT as follows:

n_classes = 20
model = load_trained_model_from_checkpoint(
  config_path,
  checkpoint_path,
  training=True,
  trainable=True,
  seq_len=SEQ_LEN,
)

# Add dense layer for classification
inputs = model.inputs[:2]
dense = model.get_layer('NSP-Dense').output
outputs = keras.layers.Dense(units=n_classes, activation='softmax')(dense)
model = keras.models.Model(inputs, outputs)

model.compile(
    RAdam(lr=LR),
    loss='sparse_categorical_crossentropy',
    metrics=['sparse_categorical_accuracy'],
)

history = model.fit(
    train_x,
    train_y,
    epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    validation_split=0.20,
    shuffle=True,
)

Here is a link to the complete tutorial with Google Colab GPU implementation for Multi-class text classification using BERT on 20 Newsgroup Dataset with Fine Tuning

Check out! https://pysnacks.com/machine-learning/bert-text-classification-with-fine-tuning/#multi-class-text-classification-using-bert

user_18408770
Related