How can I follow best practices for both project structure and logging, given that I want to run any module in any package as a standalone script?

Viewed 38

Situation

There are many resources on the web about structuring a Python project and about logging, but after hours of reading various articles (including this one), I've been unable to find any that cover enough detail as to how they work together. Given the immense flexibility of Python in relation to each topic, I imagine that it would be a monumental undertaking to comprehensively cover their interactions.

Context

The way I'm thinking about the project (assuming my strategy is reasonable), is that there are a number of standalone scripts that share various sets of utility classes and functions. I'd like to be able to run the app such that I could pick and choose to run any module within any package and/or directly as a standalone script, while leveraging logging.yaml and a way to ensure that I can always determine which module is generating the logging.

I've tried playing around with how logging works across packages and modules, depending on how I move things around, and I can sort of get everything to work as I'd like, usually with something that nags me with concerns over best practices.

Details

I'm starting with a project structure consistent with best practices (based on what I've read):

app
│   logging.yaml
│   README.md
│   requirements.txt
│   setup.py
│   
├───app
│   │   app.py
│   │   __init__.py
│   │   __main__.py
│   │   
│   └───package
│           module.py
│           __init__.py
│           
├───data
├───docs
├───logs
└───tests

Contents of logging.yaml:

version: 1
disable_existing_loggers: false
handlers:
    console:
        class: rich.logging.RichHandler
        rich_tracebacks: true
        formatter: console
        level: DEBUG
    file:
        class: logging.FileHandler
        formatter: file
        level: INFO
        filename: app.log
formatters:
    console:
        format: "%(message)s"
    file:
        format: "%(asctime)s - %(module)s, line %(lineno)d - %(name)s - %(levelname)s - %(message)s"
loggers:
    default_logger:
        handlers: [console, file]
        level: DEBUG
root:
    handlers: [console, file]
    level: DEBUG

Contents of __main__.py:

import logging
import logging.config
from datetime import datetime
from pathlib import Path

import yaml

import app


def get_logger(config_filename="logging.yaml", logs_path="logs"):

    with open(config_filename, "r") as f:
        config = yaml.safe_load(f.read())

    log_filename = Path(logs_path, f"app_{datetime.now():%Y%m%d}.log")
    config["handlers"]["file"]["filename"] = log_filename
    logging.config.dictConfig(config)

    return logging.getLogger(__name__)


if __name__ == "__main__":
    logger = get_logger()
    app.main()

Per logging best practice, I'm adding the following to each module under any imports:

logger = logging.getLogger(__name__)

When running the app, logging is working as desired and expected, but if I attempt to run any other module directly as a script, I lose my logging configuration as returned by get_logger(). Obviously I don't want to repeat that function in each of my modules, but importing it from another module makes logging appear as though it's occurring in that module (I understand why).

Any advice or suggestions as to how I can follow best practices and accomplish my aforementioned goals would be greatly appreciated!

0 Answers
Related