Getting Gensim 'TypeError: doc2bow expects an array of unicode tokens on input, not a single string' even though I am using a list?

Viewed 38

Sorry if this has been answered somewhere, but I was unable to find any examples where the person asking was actually using a list rather than a string. I totally might just be doing something dumb here, but I cannot seem to figure it out. This image shows my code - I am simply trying to initialize a dictionary in gensim using a list, but I am getting an error message saying that I am passing in a string rather than a list. I tried printing the type as well as the list itself to make sure it was in fact a list, and as you can see, it is. I also tried removing any tokens containing apostrophes in case that was for some reason causing my issues, but to no avail. I have been staring at my code forever and cannot figure out what it is so upset about. Sorry again if I am just missing something stupid, and thanks for any help you are able to provide! :)

Edit: also here is my full error message if that helps.

Edit #2: Here is my code (a bit different than before, just tried removing the punctuation).:

import gensim.corpora as corpora
import regex as re

print(type(verbs))
print(verbs)
nopunc = []

for word in verbs:
    if not re.compile(".*\'.*").match(word):
            nopunc.append(word)

print(nopunc)

id2word = corpora.Dictionary(nopunc)

Here is my error message:

Output exceeds the size limit. Open the full output data in a text editor
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~\AppData\Local\Temp\ipykernel_35400\185953347.py in <module>
     13 print(nopunc)
     14 
---> 15 id2word = corpora.Dictionary(nopunc)
     16 texts = nopunc
     17 

~\AppData\Roaming\Python\Python37\site-packages\gensim\corpora\dictionary.py in __init__(self, documents, prune_at)
     76 
     77         if documents is not None:
---> 78             self.add_documents(documents, prune_at=prune_at)
     79             self.add_lifecycle_event(
     80                 "created",

~\AppData\Roaming\Python\Python37\site-packages\gensim\corpora\dictionary.py in add_documents(self, documents, prune_at)
    202 
    203             # update Dictionary with the document
--> 204             self.doc2bow(document, allow_update=True)  # ignore the result, here we only care about updating token ids
    205 
    206         logger.info("built %s from %i documents (total %i corpus positions)", self, self.num_docs, self.num_pos)

~\AppData\Roaming\Python\Python37\site-packages\gensim\corpora\dictionary.py in doc2bow(self, document, allow_update, return_missing)
    239         """
...
--> 241             raise TypeError("doc2bow expects an array of unicode tokens on input, not a single string")
    242 
    243         # Construct (word, frequency) mapping.

TypeError: doc2bow expects an array of unicode tokens on input, not a single string
0 Answers
Related