Tf-idf with large or small corpus size

user_3261292 0
Viewed 945

"An essence of using Tf-Idf method with large corpuses is, the larger size of corpuses used is, the more unique weights terms have. This is because of the increasing of documents size in corpus or documents length gives a lower probability of duplicating a weight value for two terms in corpus. That is, the weights in Tf-Idf scheme can present a fingerprint for weights. Where in low size corpus, Tf-Idf can’t make that difference since there is huge potential of finding two terms having the same weights since they share the same source documents with the same frequency in each document. This feature can be an adversary and supporter by using Tf-Idf weighting scheme in plagiarism detection field, depending on the corpus size."

This is what I have deduced from tf-idf technique .. is it true?

Are there any link or documents can prove my conclusion؟

tf-idf
1 Answers

After 4 years of waiting for an answer, I can say the answer is yes :)

This actually can be proved simply as in the following picture. We have 4 documents and below the TF and TFIDF tables for each term.

enter image description here

When we have a small corpus (few documents), we can see that the probability that some terms have the same distribution would be high (air, quality), and because of this, their tfidf values are identical. See the table above.

But when we have a corpus with a huge amount of documents, it's less probable that we can find two terms that have the same distribution across all of the corpus.

Note: I used this website to calculate Tf-Idf: https://remykarem.github.io/tfidf-demo/

user_32612920
Related