I am attempting to use BigramCollocationFinder for a set of sentences (500 sentences).
Here is my python code logic:
words = []
#sentences = ["my name is...","i live in..."]
for sentence in sentences:
words.extend(sentence.strip().split(' '))
bigrams = nltk.collocations.BigramAssocMeasures()
finder = BigramCollocationFinder.from_words(words)
finder.apply_freq_filter(3)
outputs = finder.nbest(bigrams.likelihood_ratio, 100)
Now, I see 2 sorts of behaviour:
- If I keep the order of sentences in the 'sentences' array fixed and run the code N times, I get different number of bigrams every time
- If I change the order of sentences in the 'sentences' array and run the code N times, then also I get different number of bigrams every time
My questions:
Why is bigram finder behaving non-deterministically? Is it due the usage of likelihood_ratio?
How can it be made deterministic for a fixed input array of sentences?