I'm using logging package to log from 3 modules and multi processes in single log file, I have these three modules:
#main.py
def worker_configurer(queue):
h = logging.handlers.QueueHandler(queue)
root = logging.getLogger()
root.addHandler(h)
root.setLevel(logging.DEBUG)
def listener_configurer():
log_file_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'logging.conf')
logging.config.fileConfig(log_file_path)
def listener_process(queue, configurer):
configurer()
while True:
try:
record = queue.get()
if record is None:
break
recordName = record.name
logger = logging.getLogger(recordName)
logger.handle(record)
except Exception:
traceback.print_exc(file=sys.stderr)
if __name__=='__main__':
logging_queue = Queue(-1)
listener = Process(target=listener_process,
args=(logging_queue, listener_configurer))
listener.start()
worker_configurer(logging_queue)
logger = logging.getLogger('mylogger')
logger.info('This messgae from main process')
x = 5
y = 5
summation = a.A()
summation.toyFunc(x, y)
process = b.B(logging_queue)
process.start()
Second module, this module just log one message:
#a.py
logger = logging.getLogger('mylogger.a')
class A():
def toyFunc(self, x, y):
logger.info('result: {}'.format(x+y))
Third module, this module send message and sleep for a while:
#b.py
logger = logging.getLogger('mylogger.b')
def worker_configurer(queue):
h = logging.handlers.QueueHandler(queue)
root = logging.getLogger()
root.addHandler(h)
root.setLevel(logging.DEBUG)
class B(Process):
def __init__(self, logging_queue, func=worker_configurer):
super(B, self).__init__()
self.logging_queue = logging_queue
self.func = func
def run(self):
self.func(self.logging_queue)
for i in range(10):
logger.info('hi: {}'.format(i))
time.sleep(5)
This is my configuration file:
#logging.conf
[loggers]
keys=root,mylogger
[handlers]
keys=rootHandler,myloggerHandler
[formatters]
keys=formatter1
[logger_root]
#level=WARNING
level=ERROR
handlers=rootHandler
qualname=root
[logger_mylogger]
level=DEBUG
handlers=myloggerHandler
qualname=mylogger
propagate=0
[handler_rootHandler]
class=logging.handlers.TimedRotatingFileHandler
formatter=formatter1
args=("root.log", 'D', 1, 0, )
[handler_myloggerHandler]
class=logging.handlers.TimedRotatingFileHandler
formatter=formatter1
args=("mylogger.log", 'D', 1, 0, )
[formatter_formatter1]
format=%(asctime)s ~ %(levelname)s ~ %(process)d ~ %(module)s ~ %(funcName)s ~ %(message)s ~
This is the output:
2020-04-10 17:28:49,327 ~ INFO ~ 21820 ~ main ~ ~ This messgae from main process ~
2020-04-10 17:28:49,328 ~ INFO ~ 21820 ~ a ~ toyFunc ~ result: 10 ~
2020-04-10 17:28:49,340 ~ INFO ~ 21886 ~ b ~ run ~ hi: 0 ~
2020-04-10 17:28:49,340 ~ INFO ~ 21886 ~ b ~ run ~ hi: 0 ~
2020-04-10 17:28:54,347 ~ INFO ~ 21886 ~ b ~ run ~ hi: 1 ~
2020-04-10 17:28:54,347 ~ INFO ~ 21886 ~ b ~ run ~ hi: 1 ~
2020-04-10 17:28:59,353 ~ INFO ~ 21886 ~ b ~ run ~ hi: 2 ~
2020-04-10 17:28:59,353 ~ INFO ~ 21886 ~ b ~ run ~ hi: 2 ~
2020-04-10 17:29:04,358 ~ INFO ~ 21886 ~ b ~ run ~ hi: 3 ~
2020-04-10 17:29:04,358 ~ INFO ~ 21886 ~ b ~ run ~ hi: 3 ~
2020-04-10 17:29:09,362 ~ INFO ~ 21886 ~ b ~ run ~ hi: 4 ~
2020-04-10 17:29:09,362 ~ INFO ~ 21886 ~ b ~ run ~ hi: 4 ~
2020-04-10 17:29:14,362 ~ INFO ~ 21886 ~ b ~ run ~ hi: 5 ~
2020-04-10 17:29:14,362 ~ INFO ~ 21886 ~ b ~ run ~ hi: 5 ~
2020-04-10 17:29:19,368 ~ INFO ~ 21886 ~ b ~ run ~ hi: 6 ~
2020-04-10 17:29:19,368 ~ INFO ~ 21886 ~ b ~ run ~ hi: 6 ~
2020-04-10 17:29:24,373 ~ INFO ~ 21886 ~ b ~ run ~ hi: 7 ~
2020-04-10 17:29:24,373 ~ INFO ~ 21886 ~ b ~ run ~ hi: 7 ~
2020-04-10 17:29:29,379 ~ INFO ~ 21886 ~ b ~ run ~ hi: 8 ~
2020-04-10 17:29:29,379 ~ INFO ~ 21886 ~ b ~ run ~ hi: 8 ~
2020-04-10 17:29:34,382 ~ INFO ~ 21886 ~ b ~ run ~ hi: 9 ~
2020-04-10 17:29:34,382 ~ INFO ~ 21886 ~ b ~ run ~ hi: 9 ~
As you can see, the messages are duplicate from multiprocessing class, how to solve this problem? just to be clear, I need to call A then call B, I can't change the order, and I have to use QueueHandler to log from multi processes (this code from long and complicated code, but I made this to be simple and clear, but I got this problem in my original code). When you tried this code with Windows, no messages duplicate, but with Linux (Ubuntu 16.04), the messages are duplicate! I don't know why, I need to get same result in Windows (without duplicate). Anything to help? it would be greatly appreciated. Thanks!