Situation
There are many resources on the web about structuring a Python project and about logging, but after hours of reading various articles (including this one), I've been unable to find any that cover enough detail as to how they work together. Given the immense flexibility of Python in relation to each topic, I imagine that it would be a monumental undertaking to comprehensively cover their interactions.
Context
The way I'm thinking about the project (assuming my strategy is reasonable), is that there are a number of standalone scripts that share various sets of utility classes and functions.
I'd like to be able to run the app such that I could pick and choose to run any module within any package and/or directly as a standalone script, while leveraging logging.yaml and a way to ensure that I can always determine which module is generating the logging.
I've tried playing around with how logging works across packages and modules, depending on how I move things around, and I can sort of get everything to work as I'd like, usually with something that nags me with concerns over best practices.
Details
I'm starting with a project structure consistent with best practices (based on what I've read):
app
│ logging.yaml
│ README.md
│ requirements.txt
│ setup.py
│
├───app
│ │ app.py
│ │ __init__.py
│ │ __main__.py
│ │
│ └───package
│ module.py
│ __init__.py
│
├───data
├───docs
├───logs
└───tests
Contents of logging.yaml:
version: 1
disable_existing_loggers: false
handlers:
console:
class: rich.logging.RichHandler
rich_tracebacks: true
formatter: console
level: DEBUG
file:
class: logging.FileHandler
formatter: file
level: INFO
filename: app.log
formatters:
console:
format: "%(message)s"
file:
format: "%(asctime)s - %(module)s, line %(lineno)d - %(name)s - %(levelname)s - %(message)s"
loggers:
default_logger:
handlers: [console, file]
level: DEBUG
root:
handlers: [console, file]
level: DEBUG
Contents of __main__.py:
import logging
import logging.config
from datetime import datetime
from pathlib import Path
import yaml
import app
def get_logger(config_filename="logging.yaml", logs_path="logs"):
with open(config_filename, "r") as f:
config = yaml.safe_load(f.read())
log_filename = Path(logs_path, f"app_{datetime.now():%Y%m%d}.log")
config["handlers"]["file"]["filename"] = log_filename
logging.config.dictConfig(config)
return logging.getLogger(__name__)
if __name__ == "__main__":
logger = get_logger()
app.main()
Per logging best practice, I'm adding the following to each module under any imports:
logger = logging.getLogger(__name__)
When running the app, logging is working as desired and expected, but if I attempt to run any other module directly as a script, I lose my logging configuration as returned by get_logger(). Obviously I don't want to repeat that function in each of my modules, but importing it from another module makes logging appear as though it's occurring in that module (I understand why).
Any advice or suggestions as to how I can follow best practices and accomplish my aforementioned goals would be greatly appreciated!