How to generate BIO tags for custom entities in NER

user_3738123 0
Viewed 81

I have an annotated dataset below which contains different entities that are applicable to different sets of words in a document. Here is one of the sample document (out of 1000 documents) which contains the full text, the starting and ending position to locate an entity e.g. [426, 439, 'NO-CHEST-PAIN']

{'classes': ['FEMALE',
  'ASSOCIATED-SOB-OR-ASSOCIATED-SHORTNESS-OF-BREATH',
  'EPISODES-OF-HEART-RACING',
  'RECENT-VISIT-TO-EMERGENCY-DEPARTMENT-WITH-NEGATIVE-WORKUP',
  'NO-CHEST-PAIN',
  'NO-ILLICIT-DRUG-USE',
  'INCREASED-FREQUENCY-RECENTLY',
  'EPISODES-LAST-15-TO-30-MINUTES',
  'FEELS-HOT-OR-FEELS-CLAMMY',
  'EPISODE-OF-HAND-NUMBNESS-OR-EPISODE-OF-FINGER-NUMBNESS',
  'INCREASED-STRESS',
  '26-YEAR'],
 'annotations': [['Ms. Whelan is a 26 yo F with a 6 mo hx of heart palpitations that have been increasing in freq in the past 2 weeks. She was seen 2 weeks ago for similar symptoms with a w/u WNL. \r\n-The events last 12-30 min, are not assoc with anything particular, no alleviating or aggrevating factors. \r\n-She has assoc SOB, hot and then cold and clamy skin\r\n-Recently had numbness of fingertips on both hands during palpitation event \r\nROS: No dyspne, CP, skin, hair or wt changes. \r\nPMH: none\r\nPSH: None \r\nMeds: none \r\nAllergies: NKD\r\nFH: non contributory \r\nSH: 2 months unemployed, recently purchased a condo but is optomistic about her financial and occupational situation. Lives alone, sexual active with one partner, uses condoms. No alcohol, tobacco or drug use.',
   {'entities': [[16, 21, '26-YEAR'],
     [22, 23, 'FEMALE'],
     [48, 60, 'EPISODES-OF-HEART-RACING'],
     [76, 115, 'INCREASED-FREQUENCY-RECENTLY'],
     [120, 177, 'RECENT-VISIT-TO-EMERGENCY-DEPARTMENT-WITH-NEGATIVE-WORKUP'],
     [192, 206, 'EPISODES-LAST-15-TO-30-MINUTES'],
     [298, 307, 'ASSOCIATED-SOB-OR-ASSOCIATED-SHORTNESS-OF-BREATH'],
     [331, 336, 'FEELS-HOT-OR-FEELS-CLAMMY'],
     [357, 393, 'EPISODE-OF-HAND-NUMBNESS-OR-EPISODE-OF-FINGER-NUMBNESS'],
     [401, 418, 'EPISODES-OF-HEART-RACING'],
     [426, 439, 'NO-CHEST-PAIN'],
     [548, 567, 'INCREASED-STRESS'],
     [569, 595, 'INCREASED-STRESS'],
     [721, 752, 'NO-ILLICIT-DRUG-USE']]}]]}

On this dataset, I am further extracting BLIOU tags using spaCy

for text, annot in training_data['annotations']:
    doc=nlp(text)
    xx=([token.text for token in doc])
    yy=([token.pos_ for token in doc])
    tags=offsets_to_biluo_tags(doc, annot['entities'])
    sentence='Sentence:1'
    df = pd.DataFrame({'Sentence #':sentence,'Word': xx,'POS': yy,'Tag': tags})

which gives me an output as below:

enter image description here

Using this processed dataframe I am trying to design a BiLSTM model to predict the BIO for a given document. But the performance of the model is very abysmal due to below reasons:

  1. The number of unique target features i.e. tags is getting very large (nearly 200)
  2. The target features are highly imbalanced with 'O' tag overshadowing the other tags causing very poor predictions.

I tried to perform undersampling on the dataset (as below) but even that does not help and the validation accuracy of the model dropped to just 40%.

df_majority=df_prep.loc[df_prep['Tag'].isin(['O', '-'])]
df_minority=df_prep.loc[~df_prep['Tag'].isin(['O', '-',])]
### Now, downsamples majority labels equal to the number of samples in the minority class
df_majority = df_majority.sample(len(df_minority), random_state=0)
### concat the majority and minority dataframes
df_new = pd.concat([df_majority,df_minority])
## Shuffle the dataset to prevent the model from getting biased by similar samples
df_new = df_new.sample(frac=1, random_state=0)

Now I am wondering if there is a way to generate only the BIO tags instead so that my target feature set could be reduced. I will appreciate all suggestions that can help me solve this problem.

named-entity-recognition spacy python
1 Answers

Now I am wondering if there is a way to generate only the BIO tags instead so that my target feature set could be reduced. I will appreciate all suggestions that can help me solve this problem.

It doesn't make a difference whether you use BILOU or BIO tags as input to spaCy, they are the same internally. It also doesn't really make a difference in model performance in the general sense. This is not really something you need to worry about.

The target features are highly imbalanced with 'O' tag overshadowing the other tags causing very poor predictions.

This is completely normal for NER and you shouldn't worry about it. Consider standard NER categories like PERSON or LOCATION - most words get O and it's not a problem. If you try to replace the O tags with - so spaCy ignores them it will learn it has to label every word (so no O), which is obviously wrong.

The number of unique target features i.e. tags is getting very large (nearly 200)

This is a problem. It's hard to say with your small data sample, but some of your tags are poorly defined. 26-YEAR is not a reasonable tag - something like AGE, which covers all ages, would be reasonable. It looks like some of your tags can be grouped rather easily and some cannot. I have occasionally seen EHR annotations like this before, I would look for research on how these are handled, as I think treating these annotations as NER labels as-is just won't work. You may be able to group them or identify more general categories, and then classify them in a separate step.

user_3557150
Related