CUDA: cudaEventElapsedTime returns device not ready error

user_311148 0
Viewed 4195

I tried to measure the elapsed time on Tesla (T10 processors) and cudaEventElapsedTime returns device not ready error. But when I tested it on Fermi (Tesla M2090), it gave me the result.

Can anyone tell me what is happening...

Here is my code

cudaError_t err;
        cudaEvent_t start, stop;
        cudaEventCreate(&start);
        cudaEventCreate(&stop);
        err = cudaEventRecord(start, 0);
        f(err != cudaSuccess) {
          printf ("\n\n 1. Error: %s\n\n", cudaGetErrorString(err));
          exit(1);
        }
        // actual code
       cudaThreadSynchronize();
        err = cudaEventRecord(stop, 0);
        if(err != cudaSuccess) {
          printf ("\n\n2. Error: %s\n\n", cudaGetErrorString(err));
          exit(1);
        }
        err = cudaEventElapsedTime(&elapsed_time, start, stop);
        f(err != cudaSuccess) {
          printf ("\n\n 3. Error: %s\n\n", cudaGetErrorString(err));
          exit(1);
        }
cuda
2 Answers

Even I faced this issue and so based on the answer by @CygnusX1 I keep all my execution code in one cell and the cudaEventElapsedTime in another cell. This solved the issue because Colab (or jupyter notebook) goes to the next cell only if the process in the current cell is done.

Thus,

with torch.no_grad():
  model.eval() # warm up
  model(x)
  start.record() 
  model(x)
  model(x)
  model(x)
  end.record()
  print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/3.0))

raised the error reported in the question i.e. device not ready error and was solved by

with torch.no_grad():
  model.eval()
  model(x) # warm up
  start.record() 
  model(x)
  model(x)
  model(x)
  end.record()
# Shift the print command to next code CELL !!!

 print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/3.0))
user_130493790
Related