spaCy - Text Processing on custom model

Question

I am fairly new to machine learning and NLP in general. I am trying to wrap my head around how to do proper text pre-processing (cleaning the text).

I have built a custom text classification model, with two labels: offensive and clean. I have the below method that I run on all input text, before serving it to my model. (both before training it, and also when using it for testing).

The method will remove stopwords, punctuations and lemmatize the text.

import spacy
from spacy.lang.en.stop_words import STOP_WORDS
import string

def normalize(text, lowercase, remove_stopwords, remove_punctuation):
    nlp = spacy.load("en_core_web_sm", disable=['parser', 'tagger', 'ner'])
    stops = spacy.lang.en.stop_words.STOP_WORDS

    if lowercase:
        text = text.lower()
    text = nlp(text)
    if remove_punctuation:
        text = [t for t in text if t.text not in string.punctuation]
    lemmatized = list()
    for word in text:
        lemma = word.lemma_.strip()
        if lemma:
            if not remove_stopwords or (remove_stopwords and lemma not in stops):
                lemmatized.append(lemma)

    return " ".join(lemmatized)

Consider below input string:

Input: You're such a sweet person. All the best!

If I clean that text, using my method:

test_text = "You're such a sweet person. All the best!"
test_text = normalize(test_text, lowercase=True, remove_stopwords=True, remove_punctuation=True)

It will return: -PRON- sweet person

Now, I have tested my model with both version, and this is the result:

You're such a sweet person. All the best:

{'PROFANITY': 0.07376033067703247, 'CLEAN': 0.9841629266738892}

-PRON- sweet person

{'PROFANITY': 0.926033616065979, 'CLEAN': 0.010466966778039932}

As you can see, the results vary greatly. If I don't clean the text, before serving it to the model, it gets the profanity / clean score correct. The text is not profane.

However, if I clean the text before serving it to the model, the profanity / clean score is not correct.

Am I doing something wrong? I have a dataset with around 18k rows, that consists of labeled sentences. All sentences look like the below, and will be cleaned using my normalize method before being served to the model training:

IS_OFFENSIVE,TEXT
--------------------
1,you are a bitch!
0,you are very sweet!
0,I love you
1,"I think that is correct, idiot!"

Edit 1

This is my code for training my model.

def convert():
    TRAINING_DATA = defaultdict(list)
    # Open CSV file.
    with open('train/profanity/data/profanity_cleaned_data_cleaned.csv', mode='r') as csv_file:
        csv_reader = csv.DictReader(csv_file)
        line_count = 1
        for row in csv_reader:
            if line_count > 0 and line_count < 500:
                if row['is_offensive'] == '0':
                    CLEAN = bool(1)
                    PROFANITY = bool(0)
                else:
                    CLEAN = bool(0)
                    PROFANITY = bool(1)

                TRAINING_DATA['csv'].append([str(row['text']), {
                    "CLEAN": CLEAN, "PROFANITY": PROFANITY}])

                line_count += 1

    return TRAINING_DATA['csv']

def train():
    output_dir = 'train/profanity/model/'
    TRAINING_DATA = convert_csv_to_dataset.convert()

    nlp = spacy.blank("en")
    category = nlp.create_pipe("textcat")
    category.add_label("PROFANITY")
    category.add_label("CLEAN")
    nlp.add_pipe(category)

    # Start the training
    nlp.begin_training()

    # Loop for 10 iterations
    for itn in range(10):
        # Shuffle the training data
        random.shuffle(TRAINING_DATA)
        losses = {}

        # Batch the examples and iterate over them
        for batch in tqdm(spacy.util.minibatch(TRAINING_DATA, size=1)):
            texts = [nlp(text) for text, entities in batch]
            annotations = [{"cats": entities} for text, entities in batch]
            nlp.update(texts, annotations, losses=losses)
        # if itn % 20 == 0:
        #    print(losses)

    nlp.to_disk(output_dir)
    print("Saved model to", output_dir)

The file profanity_cleaned_data_cleaned.csv has been preprocessed using the normalize method.

user_7253453 · Answer

Looking at your normalization code, it seems you may be throwing off the model by removing so much information, and adding in elements like -PRON-

Going from

You're such a sweet person. All the best! - 10 tokens

to

-PRON- sweet person - 5 tokens (-PRON- --> - PRON -, three tokens)

Means that in the "cleaned" version, more than half of the tokens are comprised of this -PRON- text. A.k.a. the majority of the input is skewed heavily in favor of the -PRON- text, and swwet person isn't nearly as "important".

Your training code looks fine, as long as that cleaned csv is the raw input, cleaned with the same normalize function.

I would recommend the following changes

Stop including tags like -PRON- in the cleaned text
In normalize, add an else statement to the if lemma condition, where the word will be added, as is, if it doesn't have a lemma -- this may be what's causing alot of the text to get removed
Use more of your data for training, this line means you're only going to process 500 lines max, but you say you've got ~18k lines to work with if line_count > 0 and line_count < 500:
Good practice: Do not clean the text until after you read it from the csv, that way changes can be made to your normalize function without you having to re-clean and save a new csv.

spaCy - Text Processing on custom model

Edit 1

1 Answers