Extract Information with brackets using python

Question

I got a badly managed log, and need to extract into a dictionary using Python.

# Pattern: (keys are not kw1, kw2 ,etc... no pattern in key)
"para1=a, kw2=b, (b, b=b), bb, kw3=c, t4=..."

# where
# - para1=a
# - kw2=b, (b, b=b), bb
# - kw3=c
# - and so on

# extract into a dict:
out = {"para1": "a", "kw2": "b, (b, b=b), bb", "kw3": "c", "t4": ...}

# Notes several important features
'''
    1. all 'kw=value' are joined with certain spliter: ', '
    2. value and kw themselves may contain spliter. i.e. 'f(x, y)=3, f(x=3, y=2, z=1)=g(x=1, t=2)'
    3. all brackets must be in pair, (therefore we can identify spliters in kw or value).
    4. all message must be part of kw or value. 

'''

Q1: Is there a regex expression(or some Python code) that helps me get above key and value? There's no pattern in key, kwN is just a reference to key

Q2Update: Thanks to Laurent, I alr know why Q2 doesn't work: Got unexpected result. ', (.*?)=' should give me the shortest matching between ',' and '=' right?

msg = 'a, a, b=b, c=c'
re.findall(', (.*?)=', msg)
>>> ['a, b', 'c']
# I was expecting ['b','c']
# shouldn't ', (.*?)=' give me the shortest matching between ',' and '='? which is 'b' instead of 'a, b'

(New)Q3: Since I'm working with huge loads of data, working efficiency is my first priority. I've worked out a python code which could achieve the goal, but it doesnt feel quick enough, could you help me to make it better?

def my_not_efficient_solution(msg):
    '''
    Notes:
    1. all 'kw=value' are joined with certain spliter: ', '
    2. value and kw themselves may contain spliter. i.e. 'f(x, y)=3, f(x=3, y=2, z=1)=g(x=1, t=2)'
    3. all brackets must be in pair, (therefore we can identify spliters in kw or value).
    4. all message must be part of kw or value. 
    Solution:
        1. split message with spliter -> get entries
        2. check each spliter bracekt and equal sign
        3. for each entry: append to last one or serve as part of next one or good with itself
    '''
    spliter=', '
    eq_sign=['=']
    first=False
    bracket_map={'(':1,")":-1,"[":1,"]":-1}
    pair_chk_func = lambda s: not sum([bracket_map.get(i,0) for i in s])
    eq_chk_func = lambda s: sum([i in s for i in eq_sign])
    assert pair_chk_func(msg), 'msg pair check fail.'
    res = msg.split(spliter)
    
    # step1: split entry
    entries=[]
    do_pre='' # last entry is not complete(lack bracket)
    do_first = '' # last entry is not complete(lack eq sign)
    while res.__len__()>0:
        if first and entries.__len__()==2:
            entries.pop(-1)
            break
        if do_first and entries._len__()==0:
            do=do_first+res.pop(0)
        else:
            do_first=''
            do=res.pop(0)
        eq_chk=eq_chk_func(do_pre+do)
        pair_chk=pair_chk_func(do_pre+do)
        # case1: not valid entry, no eq sign
        # case2: previous entry not complete
        # case3: current entry not valid(no eq sign, will drop) and pair incomplete(may be part of next entry)
        if not eq_chk or do_pre:
            if entries.__len__() > 0:
                entries[-1]+=spliter+do
                pair_chk=pair_chk_func(entries[-1])
                if pair_chk: do_pre=''
                else: do_pre=entries[-1]
            elif not pair_chk:
                do_first=do
        # case4: current entry good to go
        elif eq_chk and pair_chk:
            entries.append(do)
            do_pre=''
        # case5: current entry not complete(pair not complete)
        else:
            entries.append(do)
            do_pre=do
    
    # step2: split each into dict
    output={}
    split_mark = '|'.join(eq_sign)
    for entry in entries:
        splits=re.split(split_mark, entry)
        if splits.__len__()<2:
            raise ValueError('split fail for message')
        kw = splits.pop(0)
        while not pair_chk_func(kw):
            kw += '='+splits.pop(0)
        output[kw]='='.join(splits)
    return output

msg = 'B_=a, kw2=b, f(A=3, k=2)=g(t=3, v=5), mark[(blabla), f(xx tt)=33]'
my_not_efficient_solution(msg)
>>> {'B_': 'a',
 'kw2': 'b',
 'f(A=3, k=2)': 'g(t=3, v=5), mark[(blabla), f(xx tt)=33]'}

user_5270581 · Answer

Answer to Q1:

Here is my suggestion:

import re
s = "kw1=a, kw2=b, (b, b=b), bb, kw3=c, kw4=..."
pattern = r'(?=(kw.)=(.*?)(?:, kw.=|$))'
result = dict(re.findall(pattern, s))
print(result) # {'kw1': 'a', 'kw2': 'b, (b, b=b), bb', 'kw3': 'c', 'kw4': '...'}

To explain the regex:

the (?=...) is a lookahead assertion to let you find overlapping matches
the ? in (.*?) makes the quantifier * (asterisk) non-greedy
the ?: makes the group (?:, kw.=|$) non-capturing
the |$ at the end allows to take account of the last value in your string

Answer to Q2:

No, this is wrong. The quantifier *? is non-greedy, so it finds the first match. Moreover there is no search for overlapping matches , which could be done with (?=...). So your observed result is the expected done. I may suggest you this simple solution:

msg = 'a, a, b=b, c=c'
result = re.findall(', ([^,]*?)=', msg)
print(result) # ['b', 'c']

user_10216112 · Answer

Q1: Is there a regex expression that helps me get above key and value?

To get the key:value in a dictionary format you can use

Say your string is

"kw1=a, kw2=b, (b, b=b), bb, kw3=c, kw4=dd, kw10=jndn"

Using the following regex gives you key and values in a list

results = re.findall(r'(\bkw\d+)=(.*?)(?=,+\s*\bkw\d+=|$)', s)

[('kw1', 'a'), ('kw2', 'b, (b, b=b), bb'), ('kw3', 'c'), ('kw4', 'dd'), ('kw10', 'jndn')]

You can convert it to a dictionary as

dict(results)

Output :

{
    'kw1': 'a', 
    'kw2': 'b, (b, b=b), bb', 
    'kw3': 'c', 
    'kw4': 'dd', 
    'kw10': 'jndn'
}

Explanation :

\b is used like a word boundary and will only match kw and not something like XYZkw
\kw\d+= Match the word kw followed by 1+ digits and =
.*? (Lazy Match) Match as least chars as possible
(?= Positive lookahead, assert to the right
- \s*\bkw\d+= Match optional whitespace chars, then pat, 1+ digits and =
- | Or
- $ Assert the end of the string for the last part
) Close the lookahead

Extract Information with brackets using python

2 Answers