how to handle lists in automatic argument parsing with argparse

user_6068294 0
Viewed 184

I've written the following module that handles arguments automatically for all my python codes, whereby I define a dictionary of my program parameters and then argparse uses this to set the command line arguments and the argument type when parsing the call sequence:

import argparse

def get_args(pars):
    parser = argparse.ArgumentParser(description='Dynamic arguments')

    # add each key of the default dictionary as an argument expecting the same type 
    for key,val in pars.items():
        parser.add_argument('--'+key,type=type(val))
    newpars=vars(parser.parse_args())

    # Missing arguments=None need to be overwritten with default value
    for key,val in newpars.items():
        if val==None:
            newpars[key]=pars[key]
    return(newpars)

if __name__ == "__main__":

    # default values:
    pars={"a":1,"animal":"cat","f":3.3}
    pars=get_args(pars)
    print(pars)

This works fine for parsing floats, integers and strings...

python3 test.py --f 2.1
{'a': 1, 'animal': 'cat', 'f': 2.1}

but now I want to add an argument that can be a list of numbers, e.g. say my default pars dictionary is this:

pars={"a":1,"animal":"cat","f":3.3,"l":[1]}

Now this happens:

python3 test.py --l [1,2,3]
{'a': 1, 'animal': 'cat', 'f': 3.3, 'l': ['[', '1', ',', '2', ',', '3', ']']}

That's not what I want, I want to get a list of numbers. In the past when I handled arguments one by one manually I would get around this by using

import ast 
val=ast.literal_eval("[1,2,3]")
[1, 2, 3]

But I can't use ast.literal_eval on each argument as it fails when not passed strings containing numerical arguments, so I thought to try this:

    try:
       parser.add_argument('--'+key,type=type(ast.literal_eval(val)))
    except:     
       parser.add_argument('--'+key,type=type(val))

and define the default list as

l="[1]"

this then returns <class 'list'> for the list argument, but how do I pick that up to conditionally use ast.literal_eval in the parsing ?

argparse python-3.x python
2 Answers

Passing an argument as a Python list is a bit "annoying" as all arguments are strings so you will need to parse it as a list. If you are willing to compromise on passing list as space-separated arguments, like --l 1 2 3, then one possible way can be to use nargs. Just add a try/except block in your loop that will check if the argument is iterable:

for key, val in pars.items():
    try:
        typ = type(val[0])
        nargs = "+"
    except TypeError:
        typ = type(val)
        nargs = None

    parser.add_argument('--'+key, nargs=nargs, type=typ)

A few notes:

  • This allows any length of list to be passed. If you want to limit the size as the list from the dictionary, just change to nargs = len(val).

  • This can only allow homogenous lists - all objects inside must be of the same type.

  • I used nargs = None and not nargs = 1 because the latter will actually make the argument a list of 1 item, while the former will pass the argument as a single object.

Embedding the above in your code (and using the second par, below for readability) will result:

pars = {"a": 1, "animal": "cat", "f": 3.3, "l": [1]}

>>> python3 test.py --f 2.1 --l 1 2
{'a': 1, 'animal': 'cat', 'f': 2.1, 'l': [1, 2]}

>>> python3 test.py --f 2.1 --l 1 2.2
test.py: error: argument --l: invalid int value: '2.2'
user_60458000

The type parameter is supposed to be a function (callable) that takes a string, and returns a value, or raises an error.

Of builtin functions, int and float are about the only useable ones:

In [40]: int("12")
Out[40]: 12
In [41]: float("12.3")
Out[41]: 12.3

Others like list or bool don't return what naive users expect:

In [42]: list("[1,2,3]")
Out[42]: ['[', '1', ',', '2', ',', '3', ']']
In [43]: bool("False")
Out[43]: True

Thus if you want to accept '--flag False' (instead of using the 'store_true' action), you need a custom function

def mybool(astr):
   if astr in ['False', 'no', 'No']:
       return False
   elif ...

The intended way of accepting lists is to use the nargs to specify how many.

$python myscript.py --foo [1,2,3] --bar a b c --baz [1 2]

is translated into sys.argv as

['myscript.py', '--foo', '[1,2,3]', '--bar', 'a', 'b', 'c', '--baz', '[1', '2]']

nargs='+' will handle the '--bar' case, but not the others.

'--foo' could be handled with type=ast.literal_eval, or maybe json.loads. But that bypasses the safeguards that argparse provides with nargs and type=int.

user_9019250
Related