Training a custom text classification model using spaCy

Question

I have below csv file called dataset.csv:

is_offensive,text
0,Hi there! My name is oliver
1,Shut up man!
0,What is wrong?
1,Go away idiot!

Which contains +5000 rows of similar annotated data.

The first column is_offensive contains the label that I want to predict, and the second column text is the actual text used for training.

After looking at the spaCy documentation, I can see that in order to train your own custom text classification model, the training data needs to look like this:

TRAINING_DATA = [
    ["Hi there! My name is oliver", {"OFFENSIVE": True}],
    ["Shup up man!", {"OFFENSIVE": True}],
    ["What is wrong?", {"OFFENSIVE": False}],
    ["Go away idiot!", {"OFFENSIVE": True}]
]

I have created the below method, to parse this CSV file and return it in the format spaCy expects:

def convert():
    TRAINING_DATA = defaultdict(list)
    # Open CSV file.
    with open('train/profanity/data/profanity_cleaned_data.csv', mode='r') as csv_file:
        csv_reader = csv.DictReader(csv_file)
        line_count = 1
        for row in csv_reader:
            if line_count > 0 and line_count < 1000: #Read the first 1000 lines.
                TRAINING_DATA['csv'].append([str(row['text']), {
                    'OFFENSIVE': bool(int(row['is_offensive']))}])
                line_count += 1

    return TRAINING_DATA['csv']

Now, to train the data, I simply do:

def train():
    output_dir = 'train/profanity/model/'
    TRAINING_DATA = convert()

    nlp = spacy.blank("en")
    category = nlp.create_pipe("textcat")
    category.add_label("OFFENSIVE")
    nlp.add_pipe(category)

    # Start the training
    nlp.begin_training()

    # Loop for 10 iterations
    for itn in range(10):
        # Shuffle the training data
        random.shuffle(TRAINING_DATA)

        # Batch the examples and iterate over them
        for batch in tqdm(spacy.util.minibatch(TRAINING_DATA, size=1)):
            texts = [nlp(text) for text, entities in batch]
            annotations = [{"cats": entities} for text, entities in batch]
            nlp.update(texts, annotations)

    nlp.to_disk(output_dir)
    print("Saved model to", output_dir)

And then finally, to test the model on a new text:

def testModel():
    test_text = "You are a very kind person."
    model_dir = 'train/profanity/model/'
    nlp = spacy.load(model_dir)
    doc = nlp(test_text)
    print(test_text, doc.cats)

The above method testModel() returns:

You are a very kind person. {'OFFENSIVE': 0.9999545812606812}

Actually, no matter what text I use as an input, I get back the OFFENSIVE label with the confidence very close to 100. However, as you can see in the input text: You are a very kind person should not be classified as OFFENSIVE

What am I doing wrong?

user_10856743 · Answer

I just encountered a similar problem. I adapted the code from Spacy V2's documentation (https://v2.spacy.io/usage/training#textcat), and like what you did, I only needed a single class, whereas the example created two classes from a single class with opposite boolean values from each other.

It turns out, there seems to be a problem if you train against a single class while also specifying classes are exclusive.

When you create the pipe, you should specify the classes are not exclusive:

nlp.create_pipe("textcat", config={"exclusive_classes": False, "architecture": "simple_cnn"})

If the "exclusive_classes" is set to True, you will need to then do the following if you have only a single class:

category.add_label("OTHER")

This 'OTHER' class is not actually used in the data labels.

Note that if you are using an evaluation method like the Spacy example linked above, you will need to add a condition to skip the 'OTHER" label (see evaluate method on link above):

for label, score in doc.cats.items():
    if label not in gold:
        continue
    if label == "OTHER":
        continue

Training a custom text classification model using spaCy

1 Answers