I have tensorflow installed, cuda installed, my program can successfully run on GPU:
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device
/job:localhost/replica:0/task:0/device:GPU:0 with 30655 MB memory:
-> device: 0, name: Tesla V100-SXM3-32GB, pci bus id: 0000:0c:00.0, compute capability: 7.0
(If it matters, I've restricted tensorflow to use just one GPU by writing the following: os.environ["CUDA_VISIBLE_DEVICES"] = "7")
I've ben trying to optimize GPU performance in hope of achieving faster fitting speeds. I was going to follow this guide on enabling XLA, i.e. to define each tf.function with jit_compile=True:
@tf.function(jit_compile=True)
def ...
however I'm stuck with this error:
2022-05-29 18:59:56.268562: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at xla_ops.cc:248 :
UNIMPLEMENTED: Could not find compiler for platform CUDA:
NOT_FOUND: could not find registered compiler for platform CUDA -- check target linkage
(hint: try adding tensorflow/compiler/jit:xla_gpu_jit as a dependency)
I'm not sure how it's possible that tensorflow can't find a compiler for CUDA, since without XLA everything works okay.
Also, I'm not really experienced in this area, so I couldn't make any sense of the "hint" (see above in the error message)
Before you ask me how's it possible that I'm noob in ML but has this powerful GPU - I don't own it, I'm just working with the Nvidia DGX machine provided by my university