How to log using MemoryHandler in Python

Viewed 24

I am very new to Python and am trying to add logs(in a log file) to a project. As soon as I added some log codes, the runtime tripled, so I was suggested to use a functionality that could perhaps store the logs in Python's memory and then print it out to a logfile - in the hope for the runtime to not increase so much.

I started searching for ways to do that and found this resource on MemoryHandler, which I interpreted as something that could help me achieve my purpose:

Base1.py (Attempt1)

I tried this code first, without the MemoryHandler:

import logging
    import logging.config
    import logging.handlers

formatter = logging.Formatter('%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:%(lineno)d — %(message)s')

def setup_logger(__name__,log_file,level=logging.INFO):
    '''to set up loggers'''
    
    handler= logging.FileHandler(log_file)
    handler.setFormatter(formatter)
    logger = logging.getLogger(__name__)
    logger.setLevel(level)
    logger.addHandler(handler)
    
    return logger

checking = setup_logger(__name__,'run_logfile.log')
checking.info("this is a test")

It took around 11 seconds to run for the entire project. This is actually a lot - because currently the data volumes are Nil and it should be around 3 seconds, which is what it was before I added these log codes. So next, I tried the below code in the hope of making the code faster:

Base2.py (Attempt2)

import logging
import logging.config
import logging.handlers

formatter = logging.Formatter('%(asctime)s — %(name)s — %(levelname)s — %(funcName)s:%(lineno)d — %(message)s')

def setup_logger(__name__,log_file,level=logging.INFO):
    '''to set up loggers'''
    
    handler= logging.FileHandler(log_file)
    handler.setFormatter(formatter)
    memoryhandler = logging.handlers.MemoryHandler(
                        capacity=1024*100,
                        flushLevel=logging.INFO,
                        target=handler,
                        flushOnClose=True
                        )
    logger = logging.getLogger(__name__)
    logger.setLevel(level)
    logger.addHandler(handler)
    logger.addHandler(memoryhandler)
    
    return logger

checking = setup_logger(__name__,'run_logfile.log')
checking.info("this is a test")

This too takes 11 seconds - it does work, but my problem is that it is not at all faster than when I did not use the MemoryHandler previously, so I was wondering if my code was very wrong still?

Am I doing anything wrong here? Or if there is way to have logs without making the runtime longer?

1 Answers

It wouldn't be faster if you're logging INFO messages and the flushLevel is also INFO - it will flush after every message. What happens if you set the flushLevel to e.g. ERROR?

Also - if adding logging triples your execution time, something else is likely to be wrong - logging doesn't add that much overhead.

Related