Which is the best way to allow configuration options be overridden at the command line in Python?

You can use ChainMap

A ChainMap groups multiple dicts or other mappings together to create a single, updateable view. If no maps are specified, a single empty dictionary is provided so that a new chain always has at least one mapping.

You can combine values from command line, environment variables, configuration file, and in case if the value is not there define a default value.

import os
from collections import ChainMap, defaultdict

options = ChainMap(command_line_options, os.environ, config_file_options,
               defaultdict(lambda: 'default-value'))
value = options['optname']
value2 = options['other-option']


print(value, value2)
'optvalue', 'default-value'

fromfile_prefix_chars

Maybe not the perfect API, but worth knowing about.

main.py

#!/usr/bin/env python3
import argparse
parser = argparse.ArgumentParser(fromfile_prefix_chars='@')
parser.add_argument('-a', default=13)
parser.add_argument('-b', default=42)
print(parser.parse_args())

Then:

$ printf -- '-a\n1\n-b\n2\n' > opts.txt
$ ./main.py
Namespace(a=13, b=42)
$ ./main.py @opts.txt
Namespace(a='1', b='2')
$ ./main.py @opts.txt -a 3 -b 4
Namespace(a='3', b='4')
$ ./main.py -a 3 -b 4 @opts.txt
Namespace(a='1', b='2')

Documentation: https://docs.python.org/3.6/library/argparse.html#fromfile-prefix-chars

Tested on Python 3.6.5, Ubuntu 18.04.

parse_args() can take an existing Namespace and merge the existing Namespace with args/options it's currently parsing; the options args/options in the "current parsing" take precedence an override anything in the existing Namespace:

foo_parser = argparse.ArgumentParser()
foo_parser.add_argument('--foo')

ConfigNamespace = argparse.Namespace()
setattr(ConfigNamespace, 'foo', 'foo')

args = foo_parser.parse_args([], namespace=ConfigNamespace)
print(args)
# Namespace(foo='foo')

# value `bar` will override value `foo` from ConfigNamespace
args = foo_parser.parse_args(['--foo', 'bar'], namespace=ConfigNamespace)
print(args)
# Namespace(foo='bar')

I've mocked it up for a real config file option. I'm parsing twice, once, as a "pre-parse" to see if the user passed a config-file, and then again for the "final parse" that integrates the optional config-file Namespace.

I have this very simple JSON config file, config.ini:

[DEFAULT]
delimiter = |

and when I run this:

import argparse
import configparser

parser = argparse.ArgumentParser()
parser.add_argument('-c', '--config-file', type=str)
parser.add_argument('-d', '--delimiter', type=str, default=',')

# Parse cmd-line args to see if config-file is specified
pre_args = parser.parse_args()

# Even if config is not specified, need empty Namespace to pass to final `parse_args()`
ConfigNamespace = argparse.Namespace()

if pre_args.config_file:
    config = configparser.ConfigParser()
    config.read(pre_args.config_file)

    for name, val in config['DEFAULT'].items():
        setattr(ConfigNamespace, name, val)


# Parse cmd-line args again, merging with ConfigNamespace, 
# cmd-line args take precedence
args = parser.parse_args(namespace=ConfigNamespace)

print(args)

with various cmd-line settings, I get:

./main.py
Namespace(config_file=None, delimiter=',')

./main.py -c config.ini
Namespace(config_file='config.ini', delimiter='|')

./main.py -c config.ini -d \;
Namespace(config_file='config.ini', delimiter=';')