Speed optimization of multiple re.match in large files

user_16280859 0
Viewed 64

Edit: Add an abstract of a file content

For multiple big files (> 10MB) stored as lists, I need to perform different match and capture and use appropriate matched data with group().

I face performance issue doing this. Using re.compile() saves me a factor 3 but it is not enough.

Here is what I do for the moment:

import re

results = [
    'EXTRACT for DC ANALYSIS',
    '  PARAM VREF =  1.0500E+00',
    '  TEMPERATURE =  2.5000E+01 Celsius',
    '  ICARLO = 9999',
    '    *VREF_INPUT =  1.0500E+00 Volts',
    '    *VREFSENSEANA =  2.1184E-01 Volts',
    '    *IREFCOMPANA =  1.7614E-05',
    '    *VOFFSET =  1.9432E-03 Volts',
    '    *IRATIO_COMP_PBIAS_DIFF__COMP_PIREFCOMP =  2.1124E+00',
    '    *IRATIO_COMP_PBIAS_OUT__COMP_PIREFCOMP =  1.0503E+00',
    '',
    'EXTRACT for DC TRANSFER CURVES',
    '  PARAM VREF =  1.0500E+00',
    '  TEMPERATURE =  2.5000E+01 Celsius',
    '  ICARLO = 10000',
    '    *VREF_INPUT =  1.0500E+00 Volts',
    '    *VREFSENSEANA =  2.1249E-01 Volts',
    '    *IREFCOMPANA =  1.6552E-05',
    '    *VOFFSET =  2.8657E-03 Volts',
    '    *IRATIO_COMP_PBIAS_DIFF__COMP_PIREFCOMP =  2.0130E+00',
    '    *IRATIO_COMP_PBIAS_OUT__COMP_PIREFCOMP =  1.0142E+00',
    '    *MC_501(VREF_INPUT) =  0.0',
    '    *MC_502(VREF_INPUT) =  1.0000E+00',
    '    *MC_600(VREF_INPUT) =  1.0500E+00',
    '    *MC_907(VREF_INPUT) = FAILED',
    '    *MC_908(VREF_INPUT) =  0.0',
]

re_analysis = re.compile(r'\s*EXTRACT for (\w+)')
re_param = re.compile(r'\s*PARAM\s+(\w+)\s*=\s*(\S+)')
re_alter = re.compile(r'\s*ALTER index (\d+)\s+(\w+)')
re_extract = re.compile(r'\s*\*(\w+)\s*=\s*(\S+)')
re_extract_mc = re.compile(r'\s*\*MC_(\d+)\((\w+)\)\s*=\s*(\S+)')
re_icarlo = re.compile(r'\s*ICARLO\s*=\s*(\d+)')

for line in results:  # self.result is the file stored as list 
    match_analysis = re_analysis.match(line)
    match_param = re_param.match(line)
    match_alter = re_alter.match(line)
    match_extract = re_extract.match(line)
    match_extract_mc = re_extract_mc.match(line)
    match_icarlo = re_icarlo.match(line)
    # do some stuff with the various match and their group()

The overall process takes ~0.5sec on a given reference file, 0.35sec being the computations of the 6 match.

I wish to strongly reduce this 0.35sec match execution time.

Is there alternate ways to 'build' the 6 matches differently in order to be faster?

Or any other ways that do not use regex that could be faster?

python-3.x regex
2 Answers

I have found a way to build one regex out of the 6 following @MegaIng suggestion to use named groups.

Unfortunately, the execution time is not so different from my initial solution, probably due to the regex complexity? So, at least with this implementation, it does not help much.

Anyway I put it here for reference, as I found the regex building quite interesting.

# sorry in advance for the eyes ...
regex = re.compile(r'\s*(?P<keyword>EXTRACT|PARAM|ALTER index|\*MC_\d+|\*|ICARLO)\s*\(*(?P<name>\w+)*\)*\s*=*\s*(?P<value>\S+)')

for line in results: 
    match = regex.match(line)
    if match:
        _reg = match.groupdict()
    else:
        continue
    # do stuff using keyword key to known which keyword is seen    
    if _reg['keyword'] == 'PARAM':
        # some stuff with name & value keys

And the groupdict() result line by line for my example list:

{'keyword': 'EXTRACT', 'name': 'for', 'value': 'DC'}
{'keyword': 'PARAM', 'name': 'VREF', 'value': '1.0500E+00'}
{'keyword': 'ICARLO', 'name': None, 'value': '9999'}
{'keyword': '*', 'name': 'VREF_INPUT', 'value': '1.0500E+00'}
{'keyword': '*', 'name': 'VREFSENSEANA', 'value': '2.1184E-01'}
{'keyword': '*', 'name': 'IREFCOMPANA', 'value': '1.7614E-05'}
{'keyword': '*', 'name': 'VOFFSET', 'value': '1.9432E-03'}
{'keyword': '*', 'name': 'IRATIO_COMP_PBIAS_DIFF__COMP_PIREFCOMP', 'value': '2.1124E+00'}
{'keyword': '*', 'name': 'IRATIO_COMP_PBIAS_OUT__COMP_PIREFCOMP', 'value': '1.0503E+00'}
{'keyword': 'EXTRACT', 'name': 'for', 'value': 'DC'}
{'keyword': 'PARAM', 'name': 'VREF', 'value': '1.0500E+00'}
{'keyword': 'ICARLO', 'name': None, 'value': '10000'}
{'keyword': '*', 'name': 'VREF_INPUT', 'value': '1.0500E+00'}
{'keyword': '*', 'name': 'VREFSENSEANA', 'value': '2.1249E-01'}
{'keyword': '*', 'name': 'IREFCOMPANA', 'value': '1.6552E-05'}
{'keyword': '*', 'name': 'VOFFSET', 'value': '2.8657E-03'}
{'keyword': '*', 'name': 'IRATIO_COMP_PBIAS_DIFF__COMP_PIREFCOMP', 'value': '2.0130E+00'}
{'keyword': '*', 'name': 'IRATIO_COMP_PBIAS_OUT__COMP_PIREFCOMP', 'value': '1.0142E+00'}
{'keyword': '*MC_501', 'name': 'VREF_INPUT', 'value': '0.0'}
{'keyword': '*MC_502', 'name': 'VREF_INPUT', 'value': '1.0000E+00'}
{'keyword': '*MC_600', 'name': 'VREF_INPUT', 'value': '1.0500E+00'}
{'keyword': '*MC_907', 'name': 'VREF_INPUT', 'value': 'FAILED'}
{'keyword': '*MC_908', 'name': 'VREF_INPUT', 'value': '0.0'}
user_162808590

I finally find a good solution using a basic split on each line, instead of regex.

Each line can be read as a sentence, making it quite easy. All needed elements of line can be captured with its index in list, or using the last index [-1]

for line in results:
    if line == '':
        continue
    _elems = line.split()

    if _elems[0].startswith('*MC_'):
        # do stuff

    # ...

    if _elems[0] == 'PARAM':
        # do stuff

The total execution time with this is 0.23 sec, with around 0.07 sec for the split stuff. Net gain of a factor of 5.

user_162808590
Related