I'm trying to build a tf idf model of type [n-gram][tfidf score] with scikit-learn. Actually I'm able to obtain n-grams with CountVectorizer() starting from a raw txt file, but when it comes to use TfIdfTransformer I don't understand why my array is empty. Here's the snippet:
with open(r"\My\Raw\document.txt") as rawdoc:
for line in rawdoc:
ngrange=(1,2)
vect = CountVectorizer(ngram_range=ngrange)
vect.fit_transform(rawdoc)
step1 = open(r"\empty\ngram_log.txt", "w")
step1.write('{1}-grams: {0}'.format(vect.get_feature_names(), ngrange))
step1.close()
rawdoc.close()
step2 = open(r"\empty\ngram_log.txt", "r") # same as before, but now is not empty anymore
pipe = Pipeline([('count', CountVectorizer()),
('tfid', TfidfTransformer())]).fit(step2)
print(pipe['count'].transform(step2).toarray()) # I'm trying to print this just so see if it's working but this print returns []
My idea is to take the raw file -> split in n-grams -> apply TfIdfTransformation -> build the matrix. Is there anything wrong?
PS When I'm in step1 if I try to use
get_feature_names_out()(which is not deprecated) it only applies on the first line of the raw doc. Why?
Thank you guys!