Could not find registered compiler for platform CUDA

Viewed 178

I have tensorflow installed, cuda installed, my program can successfully run on GPU:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device
/job:localhost/replica:0/task:0/device:GPU:0 with 30655 MB memory:
 -> device: 0, name: Tesla V100-SXM3-32GB, pci bus id: 0000:0c:00.0, compute capability: 7.0

(If it matters, I've restricted tensorflow to use just one GPU by writing the following: os.environ["CUDA_VISIBLE_DEVICES"] = "7")

I've ben trying to optimize GPU performance in hope of achieving faster fitting speeds. I was going to follow this guide on enabling XLA, i.e. to define each tf.function with jit_compile=True:

@tf.function(jit_compile=True)
def ...

however I'm stuck with this error:

2022-05-29 18:59:56.268562: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at xla_ops.cc:248 :
  UNIMPLEMENTED: Could not find compiler for platform CUDA:
     NOT_FOUND: could not find registered compiler for platform CUDA -- check target linkage
     (hint: try adding tensorflow/compiler/jit:xla_gpu_jit as a dependency)

I'm not sure how it's possible that tensorflow can't find a compiler for CUDA, since without XLA everything works okay.

Also, I'm not really experienced in this area, so I couldn't make any sense of the "hint" (see above in the error message)

Before you ask me how's it possible that I'm noob in ML but has this powerful GPU - I don't own it, I'm just working with the Nvidia DGX machine provided by my university

0 Answers
Related