Extracting dates using ML approach ( NER) in python using spacy

Viewed 412

I have the following text:

Method 1: Rule based approach using regex

text=r" this is one 2012/02/12.This is another 21-09-2021. This is third 08/09/2015 and this is final date 19870901"

import re
import pandas as pd

def extract_date_strings(text):
    extracted=re.findall(r"[0-9]+[\\\-\/\_]+[0-9]+[\\\-\/\_]+[0-9]+",text)
    return extracted    

extracted_dates=extract_date_strings(text)

def date_formatter(date_str_list):
    format_=['%Y/%m/%d','%d-%m-%Y','%d/%m/%Y']
    formatted_date=[]
    for _date, format_ in zip(extracted_dates,format_):
        date_=pd.to_datetime(_date, format=format_)
        formatted_date.append(date_)
    return pd.Series(formatted_date)

date_formatter(extracted_dates)

I get the proper result as :

0   2012-02-12
1   2021-09-21
2   2015-09-08
dtype: datetime64[ns]

Now i tried,

Method 2: ML based approach using spacy

import spacy
nlp = spacy.load('en_core_web_trf')
# nlp = spacy.load('en_core_web_lg')

doc = nlp(text)
for ent in doc.ents:
    print(ent.text, ent.label_)

I get the following result:

another 21 DATE
- DATE
09 DATE
- DATE
2021 DATE
third ORDINAL
08/09/2015 DATE
19870901 DATE

The result is not as expected (Partially successful). It is not recognizing 2012/02/12. Also it is breaking 21-09-2021 into individual elements. It recognizes successfully 08/09/2015 and 19870901.

How can i make adjustment so that spacy is able to identify date strings in text. Maybe some customization is required. Need some help here. Thanks!

1 Answers

I dealt with that problem widely, you cant use NER out of the box, it just doesnt works good enough.

I solved the problem by combining two approaches :

1.Pre processing the text using regex(I removed all punctuation so I will not be deepened on how people choose to write their dates-I was dealing with real life problem)

  1. Using Spacy pattern recognitions to detect if it is a date format

  2. Since In my case 01-mar-2000 was also a possibility I added a new NER type call months

    rulerMonths = EntityRuler( (self._nlp), overwrite_ents=True, after="ner") for Month in months: rulerMonths.add_patterns([{"label": "MONTH", "pattern": Month.lower()}])

I hope it will help you and save you time

Related