I have split text to subsequent words using ngrams with this Python script:
from nltk.util import ngrams
sentence = open('text.txt', "r")
n = 2
sixgrams = ngrams(sentence.read().split(), n)
for grams in sixgrams:
print (grams)
Now I am faced with following two problems.
1. My result looks like this
('\xd1\x8e\xd0\xbc', '\xd0\xb1\xd0\xb0\xd0\xb9\xd0\xb3\xd0\xb0\xd0\xb0\xd0\xbd')
It is maybe because text.txt file has Russian Cyrillic characters and encoded in UTF-8. Is there a way to see my result in a human readable format in Python?
2. I tried using Collections.Counter to get the each subsequent words combinations' frequency count, and print all ngrams that come up more than 2 times (sorted by value). I tried it for hours cant make it show the way I want.
input example :
Diddle diddle dumpling my son Diddle diddle my son
output example:
Diddle diddle 2
diddle dumpling 2
dumpling my 1
my son 2
son Diddle 1
Diddle my 1
my son 1
output example with limit of 2:
Diddle diddle 2
diddle dumpling 2
my son 2