Problem with determinism : set a buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable

user_13799292 0
Viewed 1074

I have problem of non-determinism with a LSTM model and I read that I should set a single buffer size in the CUBLAS_WORKSPACE_CONFIG environmental variable, with:

    CUBLAS_WORKSPACE_CONFIG=:4096:2

I am using Anaconda 1.9.12 with Python 3.7, and cudaToolkit 11.0.221. I have no clue where to insert that instruction (except that I know it is not in my python code ;-)

Any indication is most welcome. Thanks in advance for your help

cuda python
1 Answers

Some references to the CUBLAS_WORKSPACE_CONFIG environment variable are here and here.

Setting an environment variable is typically something that depends on the operating system you are using, for example windows or linux.

However, from your python script it should be possible to use a method similar to setting the CUDA_VISIBLE_DEVICES environment variable described here.

Something like this, very early in your python code:

os.environ["CUBLAS_WORKSPACE_CONFIG"]=":4096:2"

(and you may need to import os if not already)

user_16959600
Related