Handling logs in multiprocess tornado apps

user_10755628 0
Viewed 52
### Start 4 subprocesses ###
server = tornado.httpserver.HTTPServer(app)
server.bind(8000)
server.start(4) # 4 subprocesses
 
### Logger using TimeRotatingFileHandler within each app ###
timefilehandler = logging.handlers.TimedRotatingFileHandler(
    filename=os.path.join(dirname, logname + '.log'),
    when='MIDNIGHT',
    interval=1,
    encoding='utf-8'
)

Using tornado with mutiple subprocesses and logger resulted in multiple logging files subfixed like(if using file name as logging name):

service_0.log
service_1.log
service_2.log
service_3.log

Is it possible to enable all the subprocesses to write to one place in tornado? Or if it is better to use some log aggregation tools to handle the hassle since it is quite inconvenient to check the logs one by one, any ideas? Thanks in advance.

tornado logging multiprocessing python
1 Answers

You can't have different (sub)processes write to a single file - if you want to solve that you should use a log aggregator where different tornado servers log to a common endpoint (either in the cloud or locally). If you're not inclined to use a third party solution you can write one in Tornado.

Look into https://docs.python.org/3/library/logging.handlers.html to see if there's anything you like.

Or you can grep 4 files at the same time.

p.s. IIRC using subprocesses is not recommended for production, so I would suggest you run 4 processes with different ports and use the port in the log name as well.

user_150994260
Related