Remove duplicates keeping the last occurrence

Admin 1
Viewed 150

Given this haystack and Regex PCRE2(PHP>=7.3):

#1       #2      #3
green          [foo] [foo1]
red             [foo]
blue                  [foo] [foo1] [foo2]
yellow             [foo2]
green          [foo]
green          [foo] [foo1]
red             [foo]
pink                  [foo3]

Where:

#1 is always a string that can contain numbers but no spaces.

#2 is always a random amount of space between #1 and 3.

#3 same as #1 but inside of brackets [ ] and can exist multiple brackets.

I'm trying to remove all lines containing dupes on #1 but keeping the last dupe line found.

It would look like:

blue                  [foo] [foo1] [foo2]
yellow             [foo2]
green          [foo] [foo1]
red             [foo]
pink                  [foo3]

Cleared all lines that contain the same string on #1 keeping only the last.

And the lines that don't contain dupes on #1 as for example:

pink [foo3] keep them.

I tried to explain it in the most detail possible, let me know if it is still unclear or if it's not possible with regex.

regex
2 Answers

You can convert matches of the following regular expression (with flags g, m and i) to empty strings:

^([a-z\d]).*\n(?![\s\S]*\b^\1\b)

The flag g prevents returning after the first match, m (multiline) causes ^ and $ to match the beginning and end of lines rather than the beginning and end of the string, and i makes matches case insensitive.

Demo

The elements of the expression are as follows:

^             # match beginning of line
([a-z\d])     # match one or more letters or digits and save to capture group 1
.*            # match zero or more characters other than newlines
\n            # match linefeed
(?!           # begin negative lookahead
  [\s\S]*     # match zero or more characters including line terminators
  \b^\1\b     # match content of group 1 with word breaks before and after
)             # end negative lookahead

Note that . matches carriage returns \r. If the last line may not end with a line feed change \n to (?:\n|$).

If you wish to identify any strings that do not possess the required format you can use the following regular expression to match incorrectly-formatted lines:

^(?![a-z\d]*(?: *\[[^[\]\r\n]*\])+\r?\n).*

Demo

Hover your cursor over each element of the expression at the link to obtain an explanation of the function of that element.

user_2569700

You could use

^(\S+)\h+\[\S*\](?!\S).*$(?![\s\S]*^\1)
  • ^ Start of string
  • (\S+) Capture group 1
  • \h+ Match 1+ spaces
  • \[\S*\](?!\S) Match from an opening [ till closing ] and assert a whitespace boundary to the right to not match [foo]a
  • .*$ Match the rest of the line
  • (?![\s\S]*^\1) Negative lookahead, assert that capture group 1 does not occur anymore in the text

See a regex demo | PHP demo.

For example

$re = '/^(\S+)\h+\[\S*\](?!\S).*$(?![\s\S]*^\1)/m';
$str = 'green          [foo] [foo1]
red             [foo]
blue                  [foo] [foo1] [foo2]
yellow             [foo2]
green          [foo]
green          [foo] [foo1]
red             [foo]
pink                  [foo3]';

preg_match_all($re, $str, $matches);
print_r($matches[0]);

Output

Array
(
    [0] => blue                  [foo] [foo1] [foo2]
    [1] => yellow             [foo2]
    [2] => green          [foo] [foo1]
    [3] => red             [foo]
    [4] => pink                  [foo3]
)
user_54249880
Related