Can I run two Threads with their own Tensorflow session within a single python process?

Viewed 1078

I am using tensorflow 1.1 inside an nvidia docker.

I am currently trying to run two different Convolutional neural networks running inference in two seperate threads accessing the same gpu within a python process.

Firstly I am loading two different models like so:

Model 1:

with self.sess.as_default():
    saver =     tf.train.import_meta_graph('saved_models/cnn_model112.ckpt.meta')
    saver.restore(self.sess, 'saved_models/cnn_model112.ckpt')
    self.graph = self.sess.graph
with self.graph.as_default():
    self.sess.run(tf.global_variables_initializer())

Model 2:

with tf.gfile.GFile(frozen_graph_filename, "rb") as f:
graph_def = tf.GraphDef()
graph_def.ParseFromString(f.read())

with tf.Graph().as_default() as g1:
tf.import_graph_def(
        graph_def,
        input_map=None,
        return_elements=None,
        name=prefix,
        op_dict=None,
        producer_op_list=None
)
self.sess = tf.Session(graph=self.graph)

These two sessions live in separate objects. Each object spins up a thread to run inference on their associated model and targets this function:

def run(self, stop_event, streams, parent_videos, output, netType):
    if self.sess:
        with self.sess.as_default():
            while not stop_event.is_set():
                    ret, frame = stream.readSingleFrame()
                    if ret and frame is not None:
                        self.sess.run(self.predict, feed_dict={ self.input: [frame]})

This results in the following error:

2017-07-13 09:28:17.346614: E tensorflow/stream_executor/cuda/cuda_event.cc:49] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS
2017-07-13 09:28:17.346866: F tensorflow/core/common_runtime/gpu/gpu_event_mgr.cc:203] Unexpected Event status: 1

From what I read I think that multiple threads may use the same session as session.run() call is thread safe. I've also read that I could possibly merge the graphs together to run in the same session. However I would really like them to run concurrently.

I'm think the two sessions are trying to access the same GPU memory space at the same time, but I would have thought tensorflow would allocate enough room for both graphs on the gpu.

On top of my question, why does running concurrent inference on the same GPU in seperate processes work, and not in seperate threads?

0 Answers
Related