Long RegEx pattern isn't working as planned

Viewed 39

my regex pattern doesn't seem to be working in Python. This column is comma separated from a spreadsheet, between the commas there are Pipes(|) that separate things as well. I'm not worried about the pipes, however. I am needing to use the re.split() method to split the string by the comma, however, you'll note in the sample that the users input commas into the string in the first item before the first | -- hence my reason for using Regex to establish a pattern to look for. However, it's not working properly and can use another set of eyes as a beginner. I've built and run the Regex through Regex101 to help me and the explanation seems correct, but it still does not return the number of matches I expect.

My regex pattern:

".+\s\|\s\d\d\s\|\s\d\d\d\d\s\|\s\d\d\d\d\s\|\s.{2}\d\d\d\d\s\|\s\d+?\.\d+?,"gm

My Sample Test String:

ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0,ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0

Number of Matches I expect: 9 Matches

Number of Matches I get: 1 Match - (0-443) : My exported match from Regex101

"
.\s\|\s\d\d\s\|\s\d\d\d\d\s\|\s\d\d\d\d\s\|\s.\d\d\d\d\s\|\s\d\.\d,
"
gm
. matches any character (except for line terminators)
+ matches the previous token between one and unlimited times, as many times as possible, giving back as needed (greedy)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\| matches the character | literally (case sensitive)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\| matches the character | literally (case sensitive)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\| matches the character | literally (case sensitive)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\| matches the character | literally (case sensitive)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
. matches any character (except for line terminators)
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\d matches a digit (equivalent to [0-9])
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\| matches the character | literally (case sensitive)
\s matches any whitespace character (equivalent to [\r\n\t\f\v  ])
\d matches a digit (equivalent to [0-9])
+? matches the previous token between one and unlimited times, as few times as possible, expanding as needed (lazy)
\. matches the character . literally (case sensitive)
\d matches a digit (equivalent to [0-9])
, matches the character , literally (case sensitive)
Global pattern flags
g modifier: global. All matches (don't return after first match)
m modifier: multi line. Causes ^ and $ to match the begin/end of each line (not only begin/end of string)
0-443   ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 ...
Search reference
space
,
.+\s\|\s\d\d\s\|\s\d\d\d\d\s\|\s\d\d\d\d\s\|\s.{2}\d\d\d\d\s\|\s\d+?\.\d+?,
ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0,ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0
ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0,ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0
ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,```
2 Answers

Looking at the data and if you don't worry about the pipes, if you want 9 matches you can match all the values using re.findall instead of spliting and shorted the pattern a bit:

\w+:.*?\b\d+(?:\.\d+)(?=,|$)
  • \w+: Match 1+ word chars and :
  • .*? Match as least chars as possible
  • \b\d+(?:\.\d+) A word boundary, match 1+ digits an an optional decimal part
  • (?=,|$) Assert either a comma or end of the string at the right

Regex demo | Python demo

import re
from pprint import pprint

pattern = r"\w+:.*?\b\d+(?:\.\d+)(?=,|$)"
s = "ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0,ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0"

pprint(re.findall(pattern, s))

Output

['ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0',
 'ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0',
 'ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0',
 'ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0',
 'ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0',
 'ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0',
 'ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0',
 'ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0',
 'ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0']

If you have to use re.split, you can use a capture group to keep the split values and split on the comma.

The full pattern with the pipes in the match:

import re
from pprint import pprint

pattern = r"(\w+:[^|]+\|\s\d\d\s\|(?:\s\d{4}\s\|){2}\s.{2}\d{4}\s\|\s-?\d+(?:\.\d+)?),"
s = "ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0,ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0"

pprint(list(filter(None, re.split(pattern, s))))

Output

['ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0',
 'ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0',
 'ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0',
 'ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0',
 'ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0',
 'ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0',
 'ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0',
 'ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0',
 'ITSM: Laptops, Desktops, Computers | 30 | 4400 | IT0000 | 720400.0']

Python demo

Instead of matching everything at the start with a greedy match .+\s, do a lazy match .*?\s!

I also cleaned up all the duplicated \d with a quantity specifier e.g. {4}

.*?\s\|\s\d{2}\s\|\s\d{4}\s\|\s\d{4}\s\|\s.{2}\d{4}\s\|\s-?\d+?\.\d+? Demo,

which has eight matches:

0-60    ICS: Basic Maintenance | 30 | 5877 | 0000 | IT0000 | 12000.0
60-126  ,ICS: E-Rate discount (85%) | 30 | 5877 | 0000 | IT0000 | -10200.0
126-186 ,ICS: Basic Maintenance | 40 | 5877 | 0000 | IT0000 | 9000.0
186-252 ,ICMS: E-Rate discount (85%) | 40 | 5877 | 0000 | IT0000 | -7650.0
252-313 ,ICS: Basic Maintenance | 20 | 5877 | 0000 | IT0000 | 13500.0
313-379 ,ICS: E-Rate discount (85%) | 20 | 5877 | 0000 | IT0000 | -11475.0
379-442 ,ICCMS: Basic Maintenance | 70 | 5877 | 0000 | IT0000 | 12000.0
442-510 ,ICCMS: E-Rate discount (85%) | 70 | 5877 | 0000 | IT0000 | -10200.0

The last part doesn't match because there's a missing match to \s\d{4}\s. (the 0000 in bold below)

ITSM: Laptops, Desktops, Computers | 30 | 4400 | 0000 | IT0000 | 720400.0
Related