Python package hierarchy

user_5555782 0
Viewed 162

I have developed a Python package. My structure is the following :

- my_package
  - conf
  ...
  - src
    - my_package
       - main.py
         ...

For the package to perform, I need to read some configuration files from the conf directory. Initially, I was thinking about putting the conf directory outside the package to expose it to the user when he clones the package repository so that the user can easily see the files found inside it and modify them as he needs to.

Of course, the user can change the path to the configuration files so that it matches its location.

BUT I also want the package to run out of the box when installed. Therefore, inside the code, the default values are set relative to the main.py. This works in development mode.

However, when installing the package, the structure changes and becomes :

- /some/path/to/env/Lib/site-packages/my_package.egg
  - conf
  - my_package
    main.py

and I lose one level in the structure thereby breaking my relative path initially set-up. Am I doing something wrong ? I followed these principles for packaging. I could of course easily add like a dev flag which changes the way paths are treated during development mode vs production mode. But I feel I am missing something more fundamental in the packaging rules and I want to be sure.

Thanks !

python-packaging python-3.x python
1 Answers

User configuration should usually be kept in the os-specific locations for application configs. The usual way to handle it as a package author is to

  • write a default config file and put it into the package, e.g. at src/my_package/default.ini - this is better than writing the config as code, as your main.py probably does, because you can re-use your parsing step, can copy the default config to get a valid user config, and just have to maintain a single file type
  • copy the default config into a user-specific directory, use appdirs to find out which one is correct on each os
    • you can either ask your users if it's ok to create a new config at e.g. /home/user/.configs/my_app and do it after confirmation, or just do it and tell them where to find it in case they want to customize something
  • on application start, parse the default config first, then parse the user config and overwrite differing settings
  • if you also allow configuration via environment variables, those will be parsed afterwards and should overwrite both default- and user-config

Just to have a starting point, my config-parsing usually looks somewhat like this:

config.py, somewhere in your package

from configparser import ConfigParser
from pathlib import Path
import os

# 2nd party in python3.8+ only, pip-install and use importlib_resources on <3.7 
import importlib.resources  

# 3rd party, needs needs to be pip-installed
from appdirs import user_config_dir  

# a ConfigParser instance behaves like a dictionary, but has some nice utility
# like parsing .ini files, a common config file format. 
#: import this object to access the package's configuration
config = ConfigParser(interpolation=None)


def load_default_config():
    # this is how to access data files within a package 
    with importlib.resources.path("my_package", "default.ini") as default_config:
        with default_config.open() as f:
            config.read_file(f, source="default")

def load_user_config():
    user_config = Path(appdirs.user_config_dir("my_package")) / "user.ini"
    try:
        with user_config.open() as f:
            config.read_file(f, source="user")
    except FileNotFoundError:
        print(f"User config expected at '{user_config}', but not found.")


load_default_config()
load_user_config()
load_env()  # in case you need it

And to populate the initial user config, you can run something like this once:

import importlib.resources
import pathlib
import os

import appdirs


with importlib.resources.path("my_package", "default.ini") as config:
    with config.open() as f:
        content = f.read()
target = pathlib.Path(appdirs.user_config_dir("my_package")) / "user.ini"
os.makedirs(target.parent)
with open(target, "w") as f:
    f.write(content)
print(f"Created new user config at {target}.")
user_9621900
Related