I have a large corpus stored as a dict of 25 lists that I'd like to analyze with SKLearn's TfidfVectorizer. Each list contains many strings. Now, I care both about the overall term frequency (tf) throughout the whole corpus and the most unique terms within each list of the 25 strings (idf). The problem is, I haven't found a way to pass this kind of object to the TfidfVectorizer. Passing the dict just vectorizes the keys, passing the values yields an AttributeError: 'list' object has no attribute 'lower' (I guess it expects a string.)
Thanks in advance.
Update: Now including my preprocessing step, which used a dict of area, ID pairs called buckets
for area in buckets:
area_docs = []
for value in buckets[area]:
if 'filename_%s.txt' % value in os.listdir(directory):
fin = open(directory+'/filename_%s.txt' % value, 'r').read()
area_docs.append(fin)
buckets[area] = area_docs
corpus = buckets.values()
vectorizer = TfidfVectorizer(min_df=1, stop_words='english')
X = vectorizer.fit_transform(corpus)
idf = vectorizer.idf_
d = dict(zip(vectorizer.get_feature_names(), idf))
sorted_d = sorted(d.items(), key=operator.itemgetter(1))
sorted_d[:50]