Pyparsing with unstructured and unordered data

Question

I'm trying to parse unstructured data that looks like this:

ABC TTS 1382 150kt  Year 2015  US 804,4 
Landfill project Mexico RET ID 1141 Will be issued V2020 143,400t Price $124
TU/6487/Uruguay/Nuclear/2019-2020/16,000/$ 707.75
VI/707        /Spain        /Nitrate/2013        /129,465/$ 486.50

I'm aiming at extracting as much info as possible in an automatic fashion. I have defined p[yparsing expressions for each "variable" that I intend to parse and combine all these expressions into a master expression that is the "union" of all these expressions (see code below).

I got some decent recent results so far but I'm still struggling with 2 things:

How do I get pyparsing to ignore tokens for which I don't have definition (i.e. random words or numbers that I'm not interested in)
Even so, I'm using the operator & and Optional(), it seems the order of the token still matters to pyparsing. See different result that I'm getting with the 2 strings (same content, different order) that I'm trying to parse below.

Thanks in advance Kind regards Lamp'

import pyparsing as pp

# lower case / replace slash
data = [item.lower().replace("/", " ") for item in data]

# standard
standard = pp.Or(["abc","tu","vi"]).setResultsName('standard')

# price
price = pp.Regex("[0-9]+[\,|\.]?[0-9]+").set_parse_action(lambda tokens: float(tokens[0].replace(",", "."))).setResultsName('price')

# country 
country = pp.Or(["mexico", "china", "turkey", "india", "brazil"]).setResultsName('country')

# volume 
def parse_volume(toks):
  first_token = toks[0]
  if 'kt' in first_token:
    volume = float(first_token.replace("kt","")) * 1000
  elif 't' in first_token:
    volume = float(first_token.replace("t",""))
  else:
    volume = float(first_token)
  return volume

volume = pp.Regex("([0-9]+[\,|\.]?[0-9]?[k]{0,1}[t]{0,1})").set_parse_action(parse_volume).setResultsName('volume')

# quote
quote_expr = (pp.Optional(standard) & pp.Optional(country) & pp.Optional(price) & pp.Optional(volume) ).set_debug()

# examples
quote_expr.parseString("abc  china 8,4 150kt".lower()) # every get parsed
quote_expr.parseString("abc 150kt china 8,4 ".lower()) # only standard and price are parsed

user_165216 · Answer

I modified quote_expr to take an additional throwaway 'junk' term:

junk = ~country + ~standard + ~price + ~volume + pp.Word(pp.printables)

# quote
quote_expr = (pp.Optional(standard) & pp.Optional(country) & pp.Optional(price) & pp.Optional(volume) & junk[...])

To better differentiate prices from volumes, I added a negative lookahead to price:

price = pp.Regex("[0-9]+([,.][0-9]+)(?![kt])").set_parse_action(lambda tokens: float(tokens[0].replace(",", "."))).setResultsName('price')

I find it easier to use run_tests() to try different input strings:

quote_expr.runTests("""\
abc  china 8,4 150kt
abc 150kt china 8,4 
abc 150kt  sldjk china ?? 8,4 
""")

Which prints:

abc  china 8,4 150kt
['abc', 'china', 8.4, 150000.0]
- country: 'china'
- price: 8.4
- standard: 'abc'
- volume: 150000.0

abc 150kt china 8,4
['abc', 150000.0, 'china', 8.4]
- country: 'china'
- price: 8.4
- standard: 'abc'
- volume: 150000.0

abc 150kt  sldjk china ?? 8,4
['abc', 150000.0, 'sldjk', 'china', '??', 8.4]
- country: 'china'
- price: 8.4
- standard: 'abc'
- volume: 150000.0

Pyparsing with unstructured and unordered data

1 Answers