Python: Log into common file

user_5925453 0
Viewed 93

I have a python project with various nested modules. I would like to have a setup_logger function, which is called at the beginning of each file to set up the logger

from _base import setup_logger
logger = setup_logger()

and auto-generates the name of the log file after the file name of the main calling script. For example, I would like to be able to execute the script myscript.py in the terminal, and this should log to a file called myscript_<proc_id>.log. In that log file should be written by all modules that will be called during the execution. That is, my desired output is a single log file (named after the main caller, e.g. myscript_45469.log for myscript.py) and the log strings as follows:

2021-05-15 21:06:15: myscript[45469]    INFO : A log message from myscript.py
2021-05-15 21:06:16: submodule_1[45469] INFO : A log message from submodule_1.py
2021-05-15 21:06:17: submodule_2[45469] INFO : A log message from submodule_2.py

In my terminal, this is exactly what I get. However, the output is still written to separate files with only the messages of the respective module in each file:

$ ls
myscript_45469.log  submodule_1_45469.log  submodule_2_45469.log
$ cat submodule_1_45469.log
2021-05-15 21:06:16: submodule_1[45469] INFO : A log message from submodule_1.py

So far, my setup_logger function looks like follows:

# _base.py
from pathlib import Path
import logging
import os

def setup_logger(name=None, level="INFO", logfile=True):
    frame = inspect.stack()[1]
    caller_filename = frame[0].f_code.co_filename
    caller_filename = Path(caller_filename).stem

    if not name:
        name = caller_filename

    logger = logging.getLogger(name)
    logger.setLevel(os.getenv('LOGLEVEL', 'INFO').upper())

    formatter = logging.Formatter('%(asctime)s: %(name)s[%(process)d] %(levelname)-8s: %(message)s', datefmt='%Y-%m-%d %H:%M:%S')

    if logfile:
        logfile = f'{name}_{os.getpid()}.log'
        fh = logging.FileHandler(logfile)
        fh.setFormatter(formatter)
        fh.setLevel(os.getenv('LOGLEVEL', 'INFO').upper())
        logger.addHandler(fh)

    ch = logging.StreamHandler()
    ch.setFormatter(formatter)
    logger.addHandler(ch)

    return logger

The function then gets called at the beginning of each (sub)module:

# submodule_1 or submodule_2, etc
from _base import *
logger = setup_logger()

logger.info("A log message from submodule_1")

How can I achieve my desired goal without hardcoding the filename in the logger setup? I already scraped through quite some posts here on stackoverflow and many logging tutorials, but I couldn't find a solution so far. Thanks in advance for any input!

logging python
1 Answers

When you're dealing with logging, you should rely to hierarchy.

Let's assume the tree structure of your project is set like this:

project/
├─ myscript.py
├─ package/
│  ├─ submodule.py

In your submodule you will need to get the module name logger; in this case is package.submodule.

# submodule.py
import logging

logger = logging.getLogger(__name__)  # package.submodule

After that, heading to myscript, you need to setup the logger the way you want it to work.

# myscript.py
import logging

logger = logging.getLogger("package")  # intercept all "package" loggers
logger.setLevel("<your_level>")

formatter = logging.Formatter("<your_format>")

sh = logging.StreamHandler()
fh = logging.FileHandler("package.log")

for handler in (sh, fh):
    handler.setFormatter(formatter)
    logger.addHandler(handler)

Setupping your environment like this, all the loggers referenced in the execution will behave like you want them to do, so logging both to console and (same) log file, with a common format and a common log level.

If your submodule is in module-level (so there is no package), you should getLogger() - it will pick the root logger. In either case it will work as intended.

user_116470250
Related