python logging, redirecting log to file within a context

user_3133337 0
Viewed 374

How can I redirect logging within a context to a file. Also redirecting logging in other modules (3rd party e.g. requests, numpy, etc.), if called in the scope.

The use case is that we want to integrate algorithms of another team. we need to output logs of the algorithm to additional file so we can give it to them for debug purpases.

For example:

import some_func_3rd_party


some_func_3rd_party()  # logs will only be written to predefined handlers
logger.info("write only to predefined handlers")

with log2file("somefile.txt"):
    logger.info("write to file and predefined handlers")
    some_func_3rd_party()  # logs will be written to predefined handlers and to file

logger.info("write only to predefined handlers")
contextmanager python-logging python
1 Answers

At the moment, I don't see a way of achieving what you want, without accessing and modifying the root logger.

If you wanted a more targeted approach, you would need to know how the 3rd party library's logger is configured.

Please have a look at the answers to "Log all requests from the python-requests module" to better understand what I mean.

Here is one approach that might work in your particular case:

import contextlib
import logging
import requests
import sys

@contextlib.contextmanager
def special_logger(app_logger, log_file, log_level=logging.DEBUG):
    # Get all handlers added to the app_logger.
    handlers = app_logger.handlers

    # Add handlers specific to this context.
    handlers.append(logging.FileHandler(filename=log_file))
    handlers.append(logging.StreamHandler(stream=sys.stderr))

    # Get the root logger, set the logging level,
    # and add all the handlers above.
    root_logger = logging.getLogger()
    root_logger_level = root_logger.level
    root_logger.setLevel(log_level)

    for handler in handlers:
        root_logger.addHandler(handler)

    # Yield the modified root logger.
    yield root_logger

    # Clean up handlers.
    for handler in handlers:
        root_logger.removeHandler(handler)

    # Reset log level to what it was.
    root_logger.setLevel(root_logger_level)

# Get logger for this module.
app_logger = logging.getLogger('my_app')

# Add a handler logging to stdout.
sh = logging.StreamHandler(stream=sys.stdout)
app_logger.addHandler(sh)

app_logger.setLevel(logging.DEBUG)

app_logger.info("Logs go only to stdout.")

# 'requests' logs go to stdout only but won't be emitted in this case
# because root logger level is not set to DEBUG.
requests.get("http://www.google.com")

# Use the new context with modified root logger.
with special_logger(app_logger, 'my_app.log') as spec_logger:
    # The output will appear twice in the console because there is
    # one handler logging to stdout, and one to stderr. 
    # This is for demonstration purposes only.

    spec_logger.info("Logs go to stdout, stderr, and file.")

    # 'requests' logs go to stdout, stderr, and file.
    requests.get("http://www.google.com")

app_logger.info("Logs go only to stdout.")

# 'requests' logs go to stdout only but won't be emitted in this case
# because root logger level is not set to DEBUG.
requests.get("http://www.google.com")
user_152822570
Related