Regular Expression find duplicate words

user_15745095 0
Viewed 299

I have a list of words, where some are the same. I need to find these duplicates to replace them with an empty string. I don't have any problems with the replacement inside python, but with the regular expression I try to use. This is an example of my list:

AC;AC MR;TA;AC MR;AC;AC MR;TA;SDA

The words are seperatet by a semicolon.

The expression I use is:

\b(\w+\s*\w+)\b;(?=.*\1)

I tried it without the spaces between the words like that:

AC;ACMR;TA;ACMR;AC;ACMR;TA;SDA

The result I get here, when I replace the matches with nothing is:

ACMR;TA;SDA

What I want as an result in the end is:

AC;ACMR;TA;SDA

You can see that there is no AC. He always matches the standalone AC and replaces it, when it comes before the ACMR. But these are different words and I need one of every word in my result.

Can someone help me with that? Thanks in advance

duplicates python regex
4 Answers

If you want to preserve the order, one option might be using split and join and removing the spaces from the values.

s = "AC;AC MR;TA;AC MR;AC;AC MR;TA;SDA"
print(";".join([x.replace(' ', '') for x in dict.fromkeys(s.split(";"))]))

Output

AC;ACMR;TA;SDA

Python demo

If you want to find the word(s) separated by spaces, you might use re.findall with a pattern to match 1+ word chars optionally repeated by 1+ whitespace chars and 1+ word chars w+(?:\s+\w+)*

import re
s = "AC;AC MR;TA;AC MR;AC;AC MR;TA#$@#$;SDA"
print(";".join([x.replace(' ', '') for x in dict.fromkeys(re.findall(r"\w+(?:\s+\w+)*", s))]))

Output

AC;ACMR;TA;SDA

Python demo

user_54249880

This is getting much complicated than I expected, you should try a non-regex solution

Try this regex:

(?:(?<=;)|(?<=^))([^;]+)(?=.*;\1(?=;|$));
  • (?:...) non-capture group
  • (?<=;)|(?<=^) lookbehinds, make sure the following word is followed by ; or the beginning of the string
  • ([^;]+) anything that not made by ;, and put it in group 1
  • (?=.*;\1(?=;|$)) lookahead, there's a duplicate of group 1 ahead followed by a ; and a ; or the end of the string after it
  • ; match the ; after the word

Check the results

Using python:

import re

text = 'AC;AC MR;TA;AC MR;AC;AC MR;TA;SDA'
result = re.sub(r'(?:(?<=;)|(?<=^))([^;]+)(?=.*;\1(?=;|$));', '', text)
print(result) # AC;AC MR;TA;SDA
user_102892650

A regular expression doesn't seem like the right tool to use here.

Alternatively, you could split the string by its separator, get the unique values and join them back together:

s = 'AC;AC MR;TA;AC MR;AC;AC MR;TA;SDA'
print(';'.join(set(s.split(';')))) # 'AC;TA;AC MR;SDA'

Although this still involves multiple steps, it might be easier to follow than a complex reg ex. Note that the order of words may not be preserved.

user_157443000

If no use RegEx, There is a solution:

data = "AC;ACMR;TA;ACMR;AC;ACMR;TA;SDA"
dataList = data.split(";");

seen = set()
duplicated = set()
for value in dataList:  
    if value not in seen:  
        seen.add(value)
    else:
        duplicated.add(value)

print(duplicated)

Result:

{'TA', 'ACMR', 'AC'}
user_157029760
Related