Python, spaCy: Word/POS counting using text.read() and text.readlines() giving way different results

DownrightBass 0
Viewed 28

Playing with spaCy, counting the words in this Sherlock Holmes book. It was mostly a quick and clean little exercise. I ran the code this way first:

import spacy

nlp = spacy.load('en_core_web_sm')
with open("sherlock.txt", encoding='utf8') as sherlock:
    sherlock = str(sherlock.readlines())
    text = nlp(sherlock)

    count = text.count_by(spacy.attrs.POS)
    for k, v in count.items():
        print(text.vocab[k].text, v)

before deciding that changing the readlines() bit to read() would be cleaner:

import spacy

nlp = spacy.load('en_core_web_sm')
with open("sherlock.txt", encoding='utf8') as sherlock:
    sherlock = sherlock.read()
    text = nlp(sherlock)

    count = text.count_by(spacy.attrs.POS)
    for k, v in count.items():
        print(text.vocab[k].text, v)

But somehow that switch is getting me way different numbers. readlines():

X 2753
PUNCT 53127
DET 9309
NOUN 18666
ADP 11106
PROPN 8769
NUM 979
SPACE 711
ADJ 6118
PRON 15626
AUX 7491
ADV 5800
VERB 13183
PART 2265
SCONJ 3437
CCONJ 3665
INTJ 486
SYM 31

read():

DET 10211
PROPN 4048
ADP 11774
SPACE 9329
NOUN 16950
PUNCT 21603
PART 2433
PRON 16906
AUX 8133
ADV 6236
VERB 13561
ADJ 6628
CCONJ 3945
SCONJ 3735
NUM 974
INTJ 477
SYM 35
X 10

Some things like punctuation can make sense, figuring readlines() returns a list and read() returns a string. But the two result sets are 500 adjectives off from each other? 1300 pronouns? 2000 nouns? What's causing the difference, and which one do I trust? Thanks.

spacy python
1 Answers

The issue is that readlines will show a bunch of isolated lines, while read will show one lump of text with newlines in it. So imagine a sentence like this:

He saw the big dog, it was
red. He had never ever
been in a situation like that.

In a situation like this, if you saw only the second line, you wouldn't be able to tell if red is a noun or adjective, for example. In general things are going to be weird.

In that sense the read values are better, but also note spaCy's training data doesn't include newlines. While there is some whitespace augmentation to help with that (depending on your spaCy version), it's not perfect, and it'd be best to replaces newlines with spaces in a reasonable way, and maybe feed a paragraph at a time. (Also note the leftover newlines account for the large number of SPACE tags you're seeing.)

Ah, there's actually another problem.

sherlock = str(sherlock.readlines())

You shouldn't call str here, you'll get a mess. Given an input file that looks like this:

one
two
three

If you do this...

text = str(open(fname).readlines())

You get this:

['one\n', 'two\n', 'three\n']
# same as...
doc = nlp(r"['one\n', 'two\n', 'three\n']")

That kind of text format is not normal and will cause spaCy to give you many strange results.

Basically, you should check that the input to spaCy looks like normal plain text without strange formatting in order to ensure reasonable output.

polm230
Related