Ranking how direct spaCy dependencies are on tree

user_11666502 0
Viewed 312

I have a SpaCy dependency tree made by this code:

from spacy import displacy


text = "We could say to them that if in fact that's all there is, then we could, Oh, we can do something."
print(displacy.render(nlp(text), style='dep', jupyter = True, options = {'distance': 120}))

That prints out this:

enter image description here

SpaCy determines that this entire string is connected in a dependency tree. What I am trying to figure out is how to discern how direct or indirect the connection is between a word and the next word. For example, looking at the first 3 words:

  • 'We' is connected to the next word 'could', because it is directly connected to 'say', which is directly connected to 'could'. Therefor, it is 2 connection points away from the next word.
  • 'could' is directly connected to 'say'. There for it is 1 connection point away from the start.
  • and so on.

Essentially, I want to make a df that would look like this:

word  connection_points_to_next_word

We            2
could         1
say           1
...

I'm not sure how to achieve this. As SpaCy makes this graph, I'm sure there is some efficient way to calculate the number of vertices required to connect adjacent nodes, but all of SpaCy's tools I've found, such as:

Include connection information, but not how direct this connection is. Any ideas how to get closer to this problem?

spacy graph nlp python
1 Answers

Using the networkx library, we can build an undirected graph from the edgelist of token-children relationships. I am using the index of the token in the document as a unique identifier so that repeat words are treated as separate nodes.

import spacy
import networkx as nx

nlp= spacy.load('en_core_web_lg')
text = "We could say to them that if in fact that's all there is, then we could, Oh, we can do something."
doc = nlp(text)
edges = []
for tok in doc:
    edges.extend([(tok.i, child.i) for child in tok.children])

The shortest path between neighboring tokens can be calculated as below:

for idx, _ in enumerate(doc):
    if idx < len(doc)-1:
        print(doc[idx], doc[idx+1], nx.shortest_path_length(graph,source=idx, target=idx+1))

Output:

We could 2
could say 1
say to 1
to them 1
them that 4
that if 3
if in 2
in fact 1
fact that 3
that 's 1
's all 1
all there 2
there is 1
is , 4
, then 2
then we 2
we could 2
could , 2
, Oh 2
Oh , 2
, we 2
we can 2
can do 1
do something 1
something . 3
user_140563970
Related