How to get preproc for BERT

user_1496362 0
Viewed 996

I am using the stackoverflow tab classification csv dataset, which I have loaded into a dataframe: 

X = df.post
y = df.tags
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state = 42)

In addition to some other classification models, I would also like to run BERT, however, it requires a variable preproc. I am unsure which function will get that: 

import ktrain
from ktrain import text
model = text.text_classifier('bert', (x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model,train_data=(x_train, y_train), val_data=(x_test, y_test), batch_size=6)

In some documentation I see people use text.texts_from_folder(), but I have everything in a dataframe already. Is there another function in text. that woud help me get preproc?

keras python
2 Answers

See the ktrain text classification tutorial for a full list of available preprocessing functions. For instance, in your case, you can either use texts_from_df or texts_from_array. These functions will preprocess the text documents in the way expected by the model. See this example notebook for an example of using texts_from_df. Alternatively, you can use the Transformer API in ktrain.

user_135506990

I have not find too, so i wrote a function that splits csv into txt files:

import time
import os
from joblib import Parallel, delayed
from tqdm import tqdm_notebook as tqdm

treads=12
path = os.getcwd()
train_path = path + '/' + 'train_df' + '/'
test_path = path + '/' + 'test_df' + '/'

train_len = range(len(train_df['text']))
texts = train_df['text'].tolist()
ids = train_df['id'].tolist()
classes= train_df['class'].tolist()

def create_directory(directory):
    try:
        os.mkdir(directory)
    except OSError:
        print('OSError')
    else:
        print('Error')

def write_txt(text_, id_, class_, path, i):
    cur_path = path + '/' + str(id_) + '/'
    create_directory(cur_path)
    with open(cur_path + f'{class_}_{i}.txt', 'w', encoding='utf-8') as f:
        f.write(text_)

Parallel(n_jobs=treads)(delayed(write_txt)(texts[i], ids[i], classes[i], path, i) for i in tqdm(train_len))
user_98572040
Related