Enhancing the pattern
The most efficient from 5 expressions turned out to be
BodyOftheMail_Script\s=\s\[\sBEGIN\s*(\S*(?:\s++(?!END\s])\S*)*)\s+END\s]
See the regex demo
The part I modified is \S*(?:\s++(?!END\s])\S*)*:
\S* - 0 or more non-whitespace characters(?:\s++(?!END\s])\S*)* - 0 or more occurrences of
\s++(?!END\s]) - 1+ whitespace characters (matched possessively so that the lookahead check could only be performed once after all the 1+ whitespaces are matched) not followed with END, 1 whitespace and ] char\S* - 0 or more non-whitespace characters
Why not a mere BodyOftheMail_Script\s=\s\[\sBEGIN\s*(.*?)\s+END\s] with re.DOTALL? The \s*(.*?)\s+END\s] will work as follows: 0+ whitespaces will be matched at once, then (.*?) will be skipped the first time, then \s+END\s] pattern will be tried. If \s+END\s] is not matched, .*? will grab one char and again let the subsequent patterns try to match the string. And so on. It might take a lot of backtracking steps to reach the end of a match (if it is there, else, it might end in a timeout sooner than later).
Performance comparison
Since the number of steps at regex101.com is not a direct proof a certain pattern is more efficient than another, I decided to run performance tests using Python PyPi regex library. See the code below.
The results obtained on a PC with 16GB RAM, Intel Core i5-9400F CPU, consistent results are obtained using PyPi regex versions 2.5.77 and 2.5.82:
┌──────────┬─────────────────────────────────────────────────────────────────┐
│ Regex │ Time taken │
├──────────┼─────────────────────────────────────────────────────────────────┤
│ OP 1 │ 0.5606743000000001 │
│ OP 2 │ 0.5524994999999999 │
│ OP 3 │ 0.5026944 │
│ OP 4 │ 0.7502984000000001 │
│ WS_1 │ 0.25729479999999993 │
│ WS_2 │ 0.3680949 │
└──────────┴─────────────────────────────────────────────────────────────────┘
Conclusions:
- The worst OP regex is the one that contains a notorious
(.|\n)*? pattern, it is one of the most inefficient patterns I have seen in my regex life, it always causes issues across all languages. Please never use it in your patterns
- The first three OP patterns are comparable, but it is clear than the common workarounds for a
. to match any char, [\w\W] and [\s\S], should be avoided if there is a way to make . match any char with a modifier, such as (?s) or regex.DOTALL. The (?s). native solution is a tiny bit more efficient.
- My suggestion appears to be twice as fast comapring to the best OP pattern due to the fact it matches strings from left-hand delimiter to the right-hand delimiter in chunks, only stopping to check for the right-hand delimiter after grabbing whitespace chunks of text and the whitespaces that follow them.
- The
.*? construct is expanding each time a char is not the start of the right-hand delimiter, with longer strings, its efficiency will be decreasing.
The Python testing code:
import regex, timeit
text = 'BodyOftheMail_Script = [ BEGIN some text\nhere and\nhere, too \nEND ]'
regex_pattern_1=regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s{0,}((?s)[\s\S]*?)(?=\s{1,}END\s])')
regex_pattern_2=regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s{0,}((?s)[\w\W]*?)(?=\s{1,}END\s])')
regex_pattern_3=regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s{0,}((?s).*?)(?=\s{1,}END\s])')
regex_pattern_4=regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s{0,}((.|\n)*?)(?=\s{1,}END\s\])')
regex_pattern_WS_1=regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s*(\S*(?:\s++(?!END\s])\S*)*)\s+END\s]')
regexp_patternWS_2 = regex.compile(r'BodyOftheMail_Script\s=\s\[\sBEGIN\s*(.*?)\s+END\s]', regex.DOTALL)
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regex_pattern_1 as p', number=100000))
# => 0.5606743000000001
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regex_pattern_2 as p', number=100000))
# => 0.5524994999999999
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regex_pattern_3 as p', number=100000))
# => 0.5026944
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regex_pattern_4 as p', number=100000))
# => 0.7502984000000001
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regex_pattern_WS_1 as p', number=100000))
# => 0.25729479999999993
print(timeit.timeit("p.findall(text)", 'from __main__ import text, regexp_patternWS_2 as p', number=100000))
# => 0.3680949
