I am using Tensorflow and Keras. I have a python script to start the prediction process and I need to start it in a subprocess, so I do it like that:
from subprocess import Popen, PIPE, TimeoutExpired
process = Popen(['python', 'start_prediction.py'], stdout=PIPE, stderr=PIPE)
try:
output, errors = process.communicate(timeout=60)
except TimeoutExpired:
process.kill()
output, errors = process.communicate()
I have a logger configured for my application and it's working fine, but there is also a logger used by Tensorflow and it's is what I am having a problem with.
My logger is configured to use a StreamHandler with sys.stdout and a FileHandler. When I print out output and errors, all my log messages are stored in the output, there is nothing in errors. And this is what I want.
On the other hand, the logger used by Tensorflow does not work as I need it to. Some of the messages are being logged into the output (like the prediction process and some warnings) and some of them are being logged into the errors stream (other warnings). I am not able to tell if the application has failed or not, because, even if it worked and finished without any errors, I am still getting messages in the errors stream.
I tried to redirect all the messages from Tensorflow to the sys.stdout doing that:
from absl import logging
logging._warn_preinit_stderr = 0
and that
import logging
import sys
from tensorflow.python.platform import tf_logging
tf_logging.get_logger().handlers = [logging.StreamHandler(sys.stdout)]
but it does not help (logging imports are in different files, here is a simplified version). In fact, it does not make any difference, at least I did not see any.
Theoretically, I am getting warnings only, so I could suppress all warning messages and keep only errors, but I would rather not do that. There must be a better way to deal with it.
Is there a way to redirect all the Tensorflow output to the sys.stdout stream? Or maybe there is another way to not have any non-error messages in the errors output?