spacy getting tokens in the form of string instead on uint8

user_10639382 0
Viewed 161

I am wondering if there is a way to use tokenizer(s).to_array("LOWERCASE") in the form of string instead of format uint8.

from spacy.lang.en import English 
from spacy.tokenizer import Tokenizer

s = "Lets pray for the people that can be the victim of the possible eruption of Taal Volcano  keep safe everyone."

# Create nlp obj
nlp = English()
tokenizer = Tokenizer(nlp.vocab)
 
#Get a list of tokens through list comprehension
tokens = [word.text for word in tokenizer(s)]
#Out > ["Lets", "pray", "for", ... , "everyone"]

# But easier method where you can also can apply Lowercase to the tokens as well by using,
tokens = tokenizer(s).to_array("LOWER") 
#Out > array([565864407007422797, 10267499103039061205, ... ,13330460590412905967],dtype=uint64) 

#But the format you end with results is dtype Unit8

Is there a way within spacy to get this in string format?

It would make things like removing stopwords much easier

 sp = spacy.load("en_core_web_sm")
 all_stop_words = sp.Defaults.stop_words
 token_without_stopwords = [word for word in tokenizer(s).to_array("LOWER") if word not in all_stopwords]
 # This will ofcourse not work since they are two diffrent sata types from what I understand.
spacy python
2 Answers

It does not seem possible with to_array to get the string token list due to Doc.to_array return type, ndarray:

Export given token attributes to a numpy ndarray. If attr_ids is a sequence of M attributes, the output array will be of shape (N, M), where N is the length of the Doc (in tokens). If attr_ids is a single attribute, the output shape will be (N,). You can specify attributes by integer ID (e.g. spacy.attrs.LEMMA) or string name (e.g. “LEMMA” or “lemma”). The values will be 64-bit integers.

You can use

token_without_stopwords = [word for word in map(lambda x: x.text.lower(),tokenizer(s)) if word not in all_stopwords]

where map(lambda x: x.text.lower(),tokenizer(s)) gets a map object with all the token texts in lower case.

user_38329700

You can just do this.

sp = spacy.load("en_core_web_sm")
all_stop_words = sp.Defaults.stop_words
lower_words = [word.text.lower() for word in sp(s)]
filtered = [word for word in lower_words if word not in all_stopwords]
user_3557150
Related