Select rows of a Pandas DataFrame based on Spacy Rule Matcher

user_13947615 0
Viewed 487

I need to slice a pandas DataFrame based on spacy rule-based matcher results. The following is what I tried.

import pandas as pd
import numpy as np
import spacy
from spacy.matcher import Matcher

df = pd.DataFrame([['Eight people believed injured in serious SH1 crash involving truck and three cars at Hunterville',
 'Fire and emergency responding to incident at Mataura, Southland ouvea premix site',
 'Civil Defence Minister Peeni Henare heartbroken over Northland flooding',
 'Far North flooding: New photos reveal damage to roads']]).T
df.columns = ['col1']

nlp = spacy.load("en_core_web_sm")

flood_pattern = [{'LOWER': 'flooding'}]

matcher = Matcher(nlp.vocab, validate=True)
matcher.add("FLOOD_DIS", None, flood_pattern)
titles = (_ for _ in df['col1'])
g = (d for d in nlp.pipe(titles) if matcher(d))
x = list(g)

df2 = df[df['col1'].isin(x)]
df2

This produces an empty DataFrame. However, It should extract the following two rows from df.

  1. Civil Defence Minister Peeni Henare heartbroken over Northland flooding
  2. Far North flooding: New photos reveal damage to roads
pandas spacy dataframe python-3.x
2 Answers

You can do the following.

titles = (_ for _ in df['col1'])
g = (d for d in nlp.pipe(titles) if matcher(d))


A = []
for i in range(len(df)):
    doc = nlp(next(titles))
    if len(matcher(doc)) == 1:
        A.append(str(doc))
df2 = df[df['col1'].isin(A)]
user_4614360

Try this:

matcher.add("FLOOD_DIS", None, flood_pattern)
matches = [True if matcher(doc) else False for doc in nlp.pipe(df['col1'])]
df2 = df[matches][['col1']]
user_111472950
Related