Information extraction with Spacy with context awareness

user_10506754 0
Viewed 356

I'm trying to extract project relevant information via web scraping using Python+ Spacy and then building a table of projects with few attributes , example phrases that are of interest for me are:

  • The last is the 300-MW Hardin Solar III Energy Center in Roundhead, Marion, and McDonald townships in Hardin County.
  • In July, OPSB approved the 577-MW Fox Squirrel Solar Farm in Madison County.
  • San Diego agency seeking developers for pumped storage energy project.
  • The $52.5m royalty revenue-based royalty investment includes the 151MW Old Settler wind farm

Here I have highlighted different types of information that I'm interested in , I need to end up with a table with following columns : {project name} , {Location} ,{company}, {Capacity} , {start date} , {end Date} , {$investment} , {fuelType}

I'm using Spacy , but looking at the dependency tree I couldn't find any common rule , so if I use matchers I will end up with 10's of them , and they will not capture every possible information in text, is there a systematic approach that can help me achieve even a part of this task (EX: Extract capacity and assign it to the proper project name)

spacy information-extraction information-retrieval nlp python
1 Answers

You should be able to handle this with spaCy. You'll want a different strategy depending on what label you're using.

  • Location, dates, dollars: You should be able to use the default NER pipeline to get these.
  • Capacity, fuel type: You can write a simple Matcher (not DependencyMatcher) for these.
  • Company: You can use the default NER or train a custom one for this.
  • Project Name: I don't understand this from your examples. "pumped storage energy project" could be found using a Matcher or DependencyMatcher, I guess, but is hard. What are other project name examples?

A bigger problem you have is that it sounds like you want a nice neat table, but there's no guarantee your information is structured like that. What if an article mentions that a company is building two plants in the same sentence? How do you deal with multiple values? That's not a problem a library can solve for you - you have to look at your data and decide whether that doesn't happen, so you can ignore it, or what you'll do when it does happen.

user_3557150
Related