Tensorflow Warning - Insufficient Privileges: 'cupti64_101.dll'; dlerror: cupti64_101.dll

user_10227573 0
Viewed 4523

Running Tensorflow 2, Cuda 10.1 on Python 3.8 Dual GPU setup. GPUs recognised by tf2, then initally had error could not find cupti64_101.dll (a CUDA library)

After copying cupti64_101.dll into libx64 (this solution from another's question), Tensorflow can now see cupti64_101.dll, but I now receive different errors:

2020-07-31 15:31:59.563093: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1408] function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2020-07-31 15:31:59.571779: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1447] function cupti_interface_->ActivityRegisterCallbacks( AllocCuptiActivityBuffer, FreeCuptiActivityBuffer)failed with error CUPTI_ERROR_INSUFFICIENT_PRIVILEGES
2020-07-31 15:31:59.580274: E tensorflow/core/profiler/internal/gpu/cupti_tracer.cc:1430] function cupti_interface_->EnableCallback( 0 , subscriber_, CUPTI_CB_DOMAIN_DRIVER_API, cbid)failed with error CUPTI_ERROR_NOT_INITIALIZED

I changed the permissions on the cupti64_101.dll to give full permission for all applications. But still get these errors. What permission(s) are required on what file(s) to resolve these error messages please?

tensorflow cuda privileges gpu warnings
2 Answers

The privileges required relate to driver permissions. As described here, the CUPTI library requires the running user to have permission to access driver performance counters and profiling registers.

The solution for setting the permissions on Windows:

On Nvidia Control Panel, there is a Developer-Manage GPU Performance Counters section. Default toggle is to limit access to GPU preformance counters to admin users only. But you must select 'Allow acces to the GPU prformance counters to all users'. Once toggled, access permissions to the cupti dll are resolved. –

user_102275730

Adding /extras/CUPTI/lib64 to the system environment variable "path" resolved the issue for me. When i opened the visual profiler via the 'nvvp' command, it proposed this exact solution.

My CUDA path: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\extras\CUPTI\lib64

This may not directly answer the question, but belongs here very much.

I use Windows 10 and CUDA 11.4.

user_139345210
Related