How to calculate TF-IDF (using tft.tfidf function) in Tensorflow Transform

user_6244166 0
Viewed 2116

While going through the docs in tensorflow transform I came across function to perform TD-IDF.

tft.tfidf(
    x, vocab_size, smooth=True, name=None
)

As the docs in not clear in providing example of how to perform TD-IDF I tried using example_string 

example_strings=[["I", "like", "pie", "pie", "pie"], ["yum", "yum", "pie"]]

and a vocab size of 1000.(Just random number) but the below code giving me an attribute error.

tft.tfidf(example_strings, vocab_size=1000)

AttributeError: 'list' object has no attribute 'indices'

Please help me to figure this out as I am naive to Tensorflow transform ops.

tensorflow2.0 tensorflow tf-idf machine-learning nlp
1 Answers

if you would like to compute the tfidf with TFT (here an example) you can do

example_strings = ["I like pie pie pie", "yum yum pie"]
VOCAB_SIZE = 100

tf.compat.v1.disable_eager_execution()

tokens = tf.compat.v1.string_split(example_strings)
indices = tft.compute_and_apply_vocabulary(tokens, top_k=VOCAB_SIZE)
bow_indices, weight = tft.tfidf(indices, VOCAB_SIZE + 1)

otherwise, you can also use TF Tokenizer:

tk = tf.keras.preprocessing.text.Tokenizer(num_words=VOCAB_SIZE)
tk.fit_on_texts(example_strings)

tk.sequences_to_matrix(tk.texts_to_sequences(example_strings), mode='tfidf')
user_103750490
Related