Python - Locating Duplicate Words in a Text File

user_5667088 0
Viewed 3172

I was wondering if you could help me with a python programming issue? I'm currently trying to write a program that reads a text file and output "word 1 True" if the word had already occurred in that file before or "word 1 False" if this is the first time the word appeared.

Here's what I came up with:

fh = open(fname)
lst = list ()
for line in fh:
    words = line.split()
    for word in words:
        if word in words:
            print("word 1 True", word)
        else:
            print("word 1 False", word)

However, it only returns "word 1 True"

Please advise.

Thanks!

python
4 Answers

A simple (and fast) way to implement this would be with a python dictionary. These can be thought of like an array, but the index-key is a string rather than a number.

This gives some code fragments like:

found_words = {}    # empty dictionary
words1 = open("words1.txt","rt").read().split(' ')  # TODO - handle punctuation
for word in words1:
    if word in found_words:
        print(word + " already in file")
    else:
        found_words[word] = True    # could be set to anything

Now when processing your words, simply checking to see if the word already exists in the dictionary indicates that it was seen already.

user_17308950

You might also want to track previous locations, something like this:

with open(fname) as fh:
    vocab = {}
    for i, line in enumerate(fh):
       words = line.split()
       for j, word in enumerate(words):
           if word in vocab:
               locations = vocab[word]
               print word "occurs at", locations
               locations.append((i, j))
           else:
               vocab[word] = [(i, j)]
               # print "First occurrence of", word
user_4939280

This snipped code doesn't use the file, but it's easy to test and study. The main difference is that you must load the file and read per line as you did in your example

example_file = """
This is a text file example

Let's see how many time example is typed.

"""
result = {}
words = example_file.split()
for word in words:
    # if the word is not in the result dictionary, the default value is 0 + 1
    result[word] = result.get(word, 0) + 1
for word, occurence in result.items():
    print("word:%s; occurence:%s" % (word, occurence))

UPDATE:

As suggested by @khachik a better solution is using the Counter.

>>> # Find the ten most common words in Hamlet
>>> import re
>>> words = re.findall(r'\w+', open('hamlet.txt').read().lower())
>>> Counter(words).most_common(10)
[('the', 1143), ('and', 966), ('to', 762), ('of', 669), ('i', 631),
 ('you', 554),  ('a', 546), ('my', 514), ('hamlet', 471), ('in', 451)]
user_16402130

Following your route you could do this:

with open('tyger.txt', 'r') as f:
    lines = (f.read()).split()
    for word in lines:
        if lines.count(word) > 1:
            print(f"{word}: True")
        else:
            print(f"{word}: Flase")

Output

(xenial)vash@localhost:~/python/stack_overflow$ python3.7 read_true.py
When: Flase
the: True
stars: Flase
threw: Flase
down: Flase
their: True
spears: Flase
...

You could also count every word:

with open('tyger.txt', 'r') as f:
    count = {}
    lines = f.read()
    lines = lines.split()
    for i in lines:
        count[i] = lines.count(i)
    print(count)

Output

{'When': 1, 'the': 2, 'stars': 1, 'threw': 1, 'down': 1, 'their': 2,
'spears': 1, 'And': 1, "water'd": 1, 'heaven': 1, 'with': 1, 'tears:':
1, 'Did': 2, 'he': 2, 'smile': 1, 'his': 1, 'work': 1, 'to': 1,
'see?': 1, 'who': 1, 'made': 1, 'Lamb': 1, 'make': 1, 'thee?': 1}

You can use the dictionary like so:

for k in count:
    if count[k] > 1:
        print(f"{k}: True")
    else:
        print(f"{k}: False")

Ouput

When: False
the: True
stars: False
threw: False
down: False
their: True
spears: False
user_102556520
Related