How to extract BILUO tags using spaCy for conflicting entities?

user_3738123 0
Viewed 37

I am working on a Kaggle dataset and trying to extract BILUO entities using spacy

'training.offsets_to_biluo_tags'

function. The original data is in CSV format which I have managed to convert into below JSON format:

{
  "entities": [
    {
      "feature_text": "Lack-of-other-thyroid-symptoms",
      "location": "['564 566;588 600', '564 566;602 609', '564 566;632 633', '564 566;634 635']"
    },
    {
      "feature_text": "anxious-OR-nervous",
      "location": "['13 24', '454 465']"
    },
    {
      "feature_text": "Lack of Sleep",
      "location": "['289 314']"
    },
    {
      "feature_text": "Insomnia",
      "location": "['289 314']"
    },
    {
      "feature_text": "Female",
      "location": "['6 7']"
    },
    {
      "feature_text": "45-year",
      "location": "['0 5']"
    }
  ],
  "pn_history": "45 yo F. CC: nervousness x 3 weeks. Increased stress at work. Change in role from researcher to lecturer. Also many responsibilities at home, caring for elderly mother and in-laws, and 17 and 19 yo sons. Noticed decreased appetite, but forces herself to eat 3 meals a day. Associated with difficulty falling asleep (duration 30 to 60 min), but attaining full 7 hours with no interruptions, no early morning awakenings. Also decreased libido for 2 weeks. Nervousness worsened on Sunday and Monday when preparing for lectures for the week. \r\nROS: no recent illness, no headache, dizziness, palpitations, tremors, chest pain, SOB, n/v/d/c, pain\r\nPMH: none, no pasMeds: none, Past hosp/surgeries: 2 vaginal births no complications, FHx: no pysch hx, father passed from acute MI at age 65 yo, no thyroid disease\r\nLMP: 1 week ago \r\nSHx: English literature professor, no smoking, occasional EtOH, no ilicit drug use, sexually active."
}

In the JSON the entities part contains feature text and its location in the text and the pn_history part contains the entire text document.

The first problem I have is that the dataset contains instances where a single text portion is tagged with more than one unique entity. For instance, text located at position [289 314] belongs to two different entities 'Insomnia' and 'Lack of Sleep'. While processing this type of instance Spacy runs into:

ValueError [E103] Trying to set conflicting doc.ents while creating custom NER

The second problem that I have in the dataset is for some cases the starting and ending positions are clearly mentioned for instance [13 24] but there are some cases where the indices are scattered. e.g. for '564 566;588 600' which contains a semicolumn it is expected to pick the first set word(s) from the location 564 566 and the second set of word(s) from the location 588 600. These types of indexes I cannot pass to the Spacy function. Please advise how can I solve these problems.

spacy-3 named-entity-recognition python
1 Answers

OK, it sounds like you have two separate problems.

Overlapping entities. You'll need to decide what to do with these and filter your data, spaCy won't automatically handle this for you. It's up to you to decide what's "correct". Usually you would want the longest entities. You could also use the recently released spancat, which is like NER but can handle overlapping annotations.

Discontinuous entities. These are your annotations with ;. These are harder, spaCy has no way to handle them at the moment (and in my experience, few systems handle discontinuous entities). Here's an example annotation from your sample:

[no] headache, dizziness, [palpitations]

Sometimes with discontinuous entities you can just include the middle part, but that won't work here. I don't think there's any good way to translate this into spaCy, because your input tag is "lack of thyroid symptoms". Usually I would model this as "thyroid symptoms" and handle negation separately; in this case that means you could just tag palpitations.

user_3557150
Related