When having a Tensorflow dataset and multiple GPUs it is beneficial to preload data to the multiple GPUs.
The only example I could find is in this issue
host_dataset_on_cpu = .....
multi_device_iterator = prefetching_ops.MultiDeviceIterator(host_dataset_on_cpu, devices=['/gpu:0', '/gpu:1'])
elem_on_gpu_0, elem_on_gpu_1 = multi_device_iterator.get_next()
with tf.Session() as sess:
sess.run(multi_device_iterator.initializer)
elem_0 = sess.run(elem_on_gpu_0)
elem_1 = sess.run(elem_on_gpu_1)
What I am wondering is: - is it really necessary to have multiple calls to sess.run (one for each GPU?) - Can this work with a configurable number of GPUs? (splits = multi_device_iterator.get_next()) - It seems like this MultiDeviceIterator is implemented, but can not be found in the documentation. Is it still under development?
Also when I try to implement this I get a lot of errors
2018-11-02 09:22:15.740637: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.741142: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.741670: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.742512: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.748161: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.749587: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.752334: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.752756: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.753581: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.753974: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported object type list 2018-11-02 09:22:15.754434: W tensorflow/core/framework/op_kernel.cc:1261] Unimplemented: Unsupported
which seem not really helpful