Build a TF-IDF matrix from n-gram log

Viewed 23

I'm trying to build a tf idf model of type [n-gram][tfidf score] with scikit-learn. Actually I'm able to obtain n-grams with CountVectorizer() starting from a raw txt file, but when it comes to use TfIdfTransformer I don't understand why my array is empty. Here's the snippet:

with open(r"\My\Raw\document.txt") as rawdoc:

    for line in rawdoc:
        ngrange=(1,2)
        vect = CountVectorizer(ngram_range=ngrange)
        vect.fit_transform(rawdoc)

        step1 = open(r"\empty\ngram_log.txt", "w")
        step1.write('{1}-grams: {0}'.format(vect.get_feature_names(), ngrange))
        step1.close()

rawdoc.close()

step2 = open(r"\empty\ngram_log.txt", "r") # same as before, but now is not empty anymore
pipe = Pipeline([('count', CountVectorizer()), 
                    ('tfid', TfidfTransformer())]).fit(step2)

print(pipe['count'].transform(step2).toarray()) # I'm trying to print this just so see if it's working but this print returns []

My idea is to take the raw file -> split in n-grams -> apply TfIdfTransformation -> build the matrix. Is there anything wrong?

PS When I'm in step1 if I try to use get_feature_names_out() (which is not deprecated) it only applies on the first line of the raw doc. Why?

Thank you guys!

0 Answers
Related