Is there an easy way with python's logging module to send messages with a DEBUG or INFO level and the one with a higher level to different streams?
Is it a good idea anyway?
Is there an easy way with python's logging module to send messages with a DEBUG or INFO level and the one with a higher level to different streams?
Is it a good idea anyway?
Yes. You must define multiple handlers for your logging.
http://docs.python.org/library/logging.html#logging-to-multiple-destinations
http://docs.python.org/library/logging.handlers.html#module-logging.handlers
Just for your convenience adding everything together with the formatter in one package:
# shared formatter, but you can use separate ones:
FORMAT = '%(asctime)s - %(name)s - %(levelname)s - %(threadName)s - %(message)s'
formatter = logging.Formatter(FORMAT)
# single app logger:
log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
# 2 handlers for the same logger:
h1 = logging.StreamHandler(sys.stdout)
h1.setLevel(logging.DEBUG)
# filter out everything that is above INFO level (WARN, ERROR, ...)
h1.addFilter(lambda record: record.levelno <= logging.INFO)
h1.setFormatter(formatter)
log.addHandler(h1)
h2 = logging.StreamHandler(sys.stderr)
# take only warnings and error logs
h2.setLevel(logging.WARNING)
h2.setFormatter(formatter)
log.addHandler(h2)
# profit:
log.info(...)
log.debug(...)
My use case was to redirect stdout to a datafile while seeing errors on the screen during processing.
Here's the compact solution I use (tested with Python 3.10):
import logging
import sys
root_logger = logging.getLogger()
# configure default StreamHandler to stderr
logging.basicConfig(
format="%(asctime)s | %(levelname)-8s | %(filename)s:%(funcName)s(%(lineno)d) | %(message)s",
level=logging.INFO, # overall log level; DEBUG or INFO make most sense
)
stderr_handler = root_logger.handlers[0] # get default handler
stderr_handler.setLevel(logging.WARNING) # only print WARNING+
# add another StreamHandler to stdout which only emits below WARNING
stdout_handler = logging.StreamHandler(sys.stdout)
stdout_handler.addFilter(lambda rec: rec.levelno < logging.WARNING)
stdout_handler.setFormatter(stderr_handler.formatter) # reuse the stderr formatter
root_logger.addHandler(stdout_handler)
You will have to associated a handler to each stream you want logged statements to be sent to.
Python has 5 logging levels - CRITICAL, ERROR, WARNING, INFO, and DEBUG.
| Level | Numeric Value | Stream |
|---|---|---|
| CRITICAL | 50 | STDERROR |
| ERROR | 40 | STDERROR |
| WARNING | 30 | STDOUT |
| INFO | 20 | STDOUT |
| DEBUG | 10 | STDOUT |
| NOTSET | 0 |
Let's assume you want that Critical and Error logs should go to STDERROR, while others should go to STDOUT - as shown in the table above.
Here are the steps you should follow.
Step1 Initialize the logging module
import logging, sys
log = logging.getLogger()
Step 2 Create two streams, one attached to STDERROR, other to STDOUT
hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)
Step 3 Set log level for each handler. This tells handler to only log statements which have log level equal to or higher than the level we set. eg. if we set this to WARNING, the handler will only handle logging statements for WARNING, ERROR, and CRITICAL levels. Also make sure that the log level is set in handler only, and not with the log object, so that there is no conflict
hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')
Now here comes a catch. While hStErr will only output logging for ERROR and CRITICAL, hStOut will output for all 5 levels. Remember that setLevel only tells the minimum logging level which should be handled, so all levels which are greater will also be handled. To limit hStOut to not handle ERROR and CRITICAL, we use a filter.
Step 4 Specify a filter so that ERROR, and CRITICAL aren't handled by hStOut
hStOut.addFilter(lambda x : x.levelno < logging.ERROR)
Step 5 Add these handlers to logger
log.addHandler(hStErr)
log.addHandler(hStOut)
Here are all the pieces together.
import logging, sys
log = logging.getLogger()
hStErr = logging.StreamHandler(sys.stderr)
hStOut = logging.StreamHandler(sys.stdout)
hStErr.setLevel('ERROR')
hStOut.setLevel('DEBUG')
log.setLevel('NOTSET')
hStOut.addFilter(lambda x : x.levelno < logging.ERROR)
log.addHandler(hStErr)
log.addHandler(hStOut)
log.error("error log")
log.info("info log")
Output when we run this script.
error log
info log
Pycharm IDE colors output from std error red. The following image shows that the error log statement above was sent to stderr.
If we comment the addFilter line in above script, we will see the following output.
error log
error log
info log
Note that without filter hStOut will output logging statements from both INFO and ERROR, while for INFO hStErr outputs nothing, and hStOut outputs a single statement - info log
Not necessarily a good idea (it could be confusing to see info and debug messages mixed in with normal output!), but feasible, since you can have multiple handler objects and a custom filter for each of them, in order to pick and choose which log records each handler gets to handle.