Dynamically change level of python logging

Question

In my project, I have setup multiple pipelines(~20). I want to implement logging for each of these pipeline and redirect them to different file for each pipeline.

I have created a class GenericLogger as below:

class GenericLogger(object):
    def __init__(self, pipeline):
        self.name = pipeline
        pass

    def get_logger(self):
        logger = logging.getLogger(self.name)
        log_file = "{0}.log".format(self.name)
        console_handler = logging.StreamHandler()
        file_handler = logging.handlers.RotatingFileHandler(log_file, maxBytes=LOGS_FILE_SIZE, backupCount=3)
        file_format = logging.Formatter('%(asctime)s: %(levelname)s: %(name)s: %(message)s', datefmt="%Y-%m-%d %H:%M:%S")
        console_format = logging.Formatter('%(asctime)s: %(levelname)s: %(name)s: %(message)s', datefmt="%Y-%m-%d %H:%M:%S")
        console_handler.setFormatter(console_format)
        file_handler.setFormatter(file_format)
        logger.addHandler(file_handler)
        logger.addHandler(console_handler)
        logger.setLevel(logging.INFO)

        return logger

I am importing this class in my pipeline and getting the logger and using as below:

logger_helper = PythonLogger('pipeline_name')
logger = logger_helper.get_logger()
logger.warning("Something happened")

Flow of pipeline: Once triggered, they will run continuously in interval of T minutes. Currently to avoid piling up of logger objects after each complete execution I am using logger.handlers = [], and then creating a new instance of logger again on the next iteration.

Questions:

1). How Can I dynamically change the level of the logs for each pipeline separately? If I am using logging.ini, Is creating a static handlers/formatters for each pipeline is necessary or Is there something I can do dynamically? don't know much about this.

2). Is the above implementation of the logger is correct or creating a Class for logger is something which should not be done?

user_118903 · Answer

To answer your two points:

You could have a mapping between pipeline name and level, and after creating the logger for a pipeline, you can set its level appropriately.
You don't need multiple console handlers, do you? I'd just create one console handler and attach it to the root logger. Likewise, you don't need to create multiple identical file and console formatters - just make one of each. In fact, since they are all apparently identical format strings, you just need one formatter instance. Avoid creating a class like GenericLogger.

Thus, something like:

formatter = logging.Formatter(...)
console_handler = logging.StreamHandler()
console_handler.setFormatter(formatter)
logging.getLogger().addHandler(console_handler)
for name, level in name_to_level.items():
    logger = logging.getLogger(name)
    logger.setLevel(level)
    file_handler = logging.handlers.RotatingFileHandler(...)
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)

should do what you need.

Dynamically change level of python logging

1 Answers