Filtering data based on NER entity labels in spacy training data

Admin 1
Viewed 172

I have an NER training data using Spacy in the following format.

[('Christmas Perot 2021 TSO\nSkip to Main Content HOME CONCERTS EVENTS ABOUT STAFF EDUCATION SUPPORT US More Use tab to navigate through the menu items. BUY TICKETS SUNDAY, DECEMBER 12, 2021 I PEROT THEATRE I 4:00 PM\nPOPS I Christmas at The Perot\nCLICK HERE to purchase tickets, or contact the Texarkana Symphony Orchestra at 870.773.3401\nA Texarkana Tradition Join the TSO, the Texarkana Jazz Orchestra, and the TSO Chamber Singers, for this holiday concert for the whole family.\nDon’t miss seeing the winner of TSO’s 11th Annual Celebrity Conductor Competition\nBack to Events 2019 Texarkana Symphony Orchestra',
  {'entities': [(375, 399, 'organization'),
    (290, 318, 'organization'),
    (220, 242, 'production_name'),
    (169, 186, 'performance_date'),
    (189, 202, 'auditorium'),
    (205, 212, 'performance_starttime'),
    (409, 428, 'organization')]})]

Data is the first element in the tuple. Within entities, the numbers represent the character position (start and end) of entities in data. Some lines do not have any entities. For example first line Christmas Perot 2021 TSO do not have any entities. I need to remove sentences which do not have any entities. Removing of sentences can be done based in . and \n characters. I got the entity data based on the character number but i didnt manage to get the removal of sentences which are not tagged

Code

from tqdm import tqdm
import spacy
nlp = spacy.blank("en") # load a new spacy model
db = DocBin() # create a DocBin object
for text, annot in tqdm(train_data): # data in previous format
    doc = nlp.make_doc(text) # create doc object from text
    ents = []
    for start, end, label in annot["entities"]: # add character indexes
        span = doc.char_span(start, end, label=label, alignment_mode="contract")
        print(start,end,span,label)
        if span is None:
            print("Skipping entity")
        else:
            ents.append(span)
    doc.ents = ents # label the text with the ents
spacy python
1 Answers

How about this:

#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
import numpy as np

foo = \
    [('''Christmas Perot 2021 TSO
Skip to Main Content HOME CONCERTS EVENTS ABOUT STAFF EDUCATION SUPPORT US More Use tab to navigate through the menu items. BUY TICKETS SUNDAY, DECEMBER 12, 2021 I PEROT THEATRE I 4:00 PM
POPS I Christmas at The Perot
CLICK HERE to purchase tickets, or contact the Texarkana Symphony Orchestra at 870.773.3401
A Texarkana Tradition Join the TSO, the Texarkana Jazz Orchestra, and the TSO Chamber Singers, for this holiday concert for the whole family.
Don\xe2\x80\x99t miss seeing the winner of TSO\xe2\x80\x99s 11th Annual Celebrity Conductor Competition
Back to Events 2019 Texarkana Symphony Orchestra''',
     {'entities': [
    (375, 399, 'organization'),
    (290, 318, 'organization'),
    (220, 242, 'production_name'),
    (169, 186, 'performance_date'),
    (189, 202, 'auditorium'),
    (205, 212, 'performance_starttime'),
    (409, 428, 'organization'),
    ]})]

print(foo[0][0])
sentences = re.split(r'\.|\n', foo[0][0])
sentence_lengths = list(map(len, sentences))

cumulative_sentence_length = np.cumsum(sentence_lengths) - 1

pick_indices = set()

entities = foo[0][1]['entities']

for e in entities:
    # only pick the first index (→ second [0])
    idx = np.where(e[0] < cumulative_sentence_length)[0][0]
    print('\n\nIndex:', idx, 'Entity:', e, 'Range:', [
        [0, *cumulative_sentence_length][idx],
        [0, *cumulative_sentence_length][idx+1]
    ], '\nSentence:', sentences[idx])
    pick_indices.add(idx)

print(pick_indices)
print('\n'.join([sentences[i] for i in pick_indices]))

The output is the first, second, third and fourth (= {2, 3, 4, 7}) sentence. The idea is to

  1. split the sentences
  2. cumulate the length of the sentence
  3. check if the entity start index is within range (and exclusively pick the first index)
  4. (optional) you can do a sanity check yourself with the end index of your entity

Have a look at the cumulative_sentence_length variable which holds the value [ 23 145 209 238 320 323 327 467 467 552 600] which are the upper bounds for the sentence intervals.

As you are dealing with a data science topic I presume that the use of numpy here is no hurdle for you.

user_24022810
Related