(As a student I am kind of new to this but did quite a bit of research and I got pretty far, I'm super into learning something new through this!)
This issue is for the project pulse -> https://github.com/adamian98/pulse
the readme if you scroll down a bit on the page, gives a much better explanation than I could. It will also give a direct "correct" path to judge my actions against and make solving the problem a lot easier.
Objective: run program using the run.py file
Issue: I got a "RuntimeError: CUDA out of memory" despite having a compatible gpu and enough vram
Knowledge: when it comes to coding i just started a few days ago and have a dozen hours with anaconda now, comfterable creating environments.
What I did was... (the list below is a summary and the specific details are after it)
install anaconda
use this .yml file -> https://github.com/leihuayi/pulse/blob/feature/docker/pulse.yml (it changes dependencies to work for windows which is why I needed to grab a different one than the one supplied on the master github page) to create a new environment and install the required packages. It worked fantastically! I only got an error trying to install dlib, it didn't seem compatible with A LOT of the packages and my python version.
I installed the cuda toolkit 10.2 , cmake 3.17.2, and tried to install dlib into the environment directly. the errors spat out in a blaze of glory. The dlib package seems to be only needed for a different .py file and not run.py though so I think it may be unrelated to this error
logs are below and I explain my process in more detail
START DETAILS AND LOGS: from here until the "DETAILS 2" section should be enough information to solve, the rest past there is in case
error log for runing out of memory--> (after executing the "run.py" file)
Loading Synthesis Network
Loading Mapping Network
Running Mapping Network
Traceback (most recent call last):
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\run.py", line 58, in
model = PULSE(cache_dir=kwargs["cache_dir"])
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\PULSE.py", line 44, in init
latent_out = torch.nn.LeakyReLU(5)(mapping(latent))
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 233, in forward
x = super().forward(x)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
input = module(input)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 38, in forward
return F.linear(x, self.weight * self.w_mul, bias)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)
End of error log.
NVIDIA-SMI LOG WHILE RUNNING (checking free memory)
C:\Users\micha>nvidia-smi --query-gpu=memory.free --format=csv --loop=1
memory.free [MiB]
5991 MiB 5991 MiB 5991 MiB 5991 MiB 5897 MiB 5781 MiB 5685 MiB 1643 MiB 5991 MiB 5991 MiB
the program stopped at the 1643MiB
DETAILS PART 1:
I have an nvidia gpu with 6gb of memory which, (according to other logs posted by the creator of the project-> should have enough memory for this to work "We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well" -adamian98 ). I'm trying to fix that error and get the run.py to work as intended.
Here is the system info using the numba -s command in anaconda (includes hardware info: gpu, my windows version, memory, python version etc.)
START CUDA INFO
Hardware Information Machine : AMD64 CPU Name : znver1 CPU Count : 16 Number of accessible CPUs : 16 List of accessible CPUs cores : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 CFS Restrictions (CPUs worth of runtime) : None CPU Features : 64bit adx aes avx avx2 bmi bmi2 clflushopt clwb clzero cmov cx16 cx8 f16c fma fsgsbase fxsr lzcnt mmx movbe mwaitx pclmul popcnt prfchw rdpid rdrnd rdseed sahf sha sse sse2 sse3 sse4.1 sse4.2 sse4a ssse3 wbnoinvd xsave xsavec xsaveopt xsaves Memory Total (MB) : 15789 Memory Available (MB) : 8421 OS Information Platform Name : Windows-10-10.0.18362-SP0 Platform Release : 10 OS Name : Windows OS Version : 10.0.18362 OS Specific Version : 10 10.0.18362 SP0 Multiprocessor Free Libc Version : ? Python Information Python Compiler : MSC v.1916 64 bit (AMD64) Python Implementation : CPython Python Version : 3.8.5 Python Locale : en_CA.cp1252 LLVM Information LLVM Version : 10.0.1 CUDA Information CUDA Device Initialized : True CUDA Driver Version : 10020 CUDA Detect Output: Found 1 CUDA devices id 0 b'GeForce GTX 1660 Ti with Max-Q Design' [SUPPORTED] compute capability: 7.5 pci device id: 0 pci bus id: 1 Summary: 1/1 devices are supported CUDA Librairies Test Output: Finding cublas from named cublas.dll trying to open library... ERROR: failed to open cublas: Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cusparse from named cusparse.dll trying to open library... ERROR: failed to open cusparse: Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cufft from named cufft.dll trying to open library... ERROR: failed to open cufft: Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding curand from named curand.dll trying to open library... ERROR: failed to open curand: Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding nvvm from named nvvm.dll trying to open library... ERROR: failed to open nvvm: Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding cudart from named cudart.dll trying to open library... ERROR: failed to open cudart: Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax. Finding libdevice from searching for compute_20... ERROR: can't open libdevice for compute_20 searching for compute_30... ERROR: can't open libdevice for compute_30 searching for compute_35... ERROR: can't open libdevice for compute_35 searching for compute_50... ERROR: can't open libdevice for compute_50 ROC information ROC Available : False ROC Toolchains : None HSA Agents Count : 0 HSA Agents: None HSA Discrete GPUs Count : 0 HSA Discrete GPUs : None SVML Information SVML State, config.USING_SVML : True SVML Library Loaded : True llvmlite Using SVML Patched LLVM : True SVML Operational : True Threading Layer Information TBB Threading Layer Available : False +--> Disabled due to Unknown import problem. OpenMP Threading Layer Available : True +-->Vendor: MS Workqueue Threading Layer Available : True +-->Workqueue imported successfully. Numba Environment Variable Information None found. Conda Information Conda Build : 3.20.5 Conda Env : 4.9.2 Conda Platform : win-64 Conda Python Version : 3.8.5.final.0 Conda Root Writable : True Installed Packages blas 1.0 mkl ca-certificates 2020.12.8 haa95532_0 certifi 2020.12.5 py38haa95532_0 cffi 1.14.0 py38h7a1dbc1_0 chardet 3.0.4 py38haa95532_1003 cryptography 2.9.2 py38h7a1dbc1_0 cudatoolkit 10.2.89 h74a9793_1 anaconda cycler 0.10.0 py38_0 freetype 2.9.1 ha9979f8_1 icc_rt 2019.0.0 h0cc432a_1 icu 58.2 ha925a31_3 idna 2.9 py_1 intel-openmp 2019.4 245 jpeg 9b hb83a4c4_2 kiwisolver 1.2.0 py38h74a9793_0 libcxx 7.0.0 h1ad3211_1002 conda-forge libpng 1.6.37 h2a8f88b_0 libtiff 4.1.0 h56a325e_0 llvm-meta 7.0.0 0 conda-forge m2-bash 4.3.042 5 m2-gcc-libs 5.3.0 4 m2-libedit 3.1 20150326 m2-libffi 3.2.1 2 m2-libreadline 6.3.008 8 m2-msys2-runtime 2.5.0.17080.65c939c 3 m2-ncurses 6.0.20160220 2 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs-core 5.3.0 7 m2w64-gmp 6.1.0 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 matplotlib 3.1.3 py38_0 matplotlib-base 3.1.3 py38h64f37c6_0 mkl 2019.4 245 mkl-service 2.3.0 py38h196d8e1_0 mkl_fft 1.0.15 py38h14836fe_0 mkl_random 1.1.0 py38hf9181ef_0 msys2-conda-epoch 20160418 1 ninja 1.9.0 py38h74a9793_0 numpy 1.18.1 py38h93ca92e_0 numpy-base 1.18.1 py38hc3f5095_1 olefile 0.46 py_0 openssl 1.1.1i h2bbff1b_0 pandas 1.0.3 py38h47e9c7a_0 pillow 7.1.2 py38hcc1f983_0 pip 20.0.2 py38_3 powershell_shortcut 0.0.1 3 pycparser 2.20 py_2 pyopenssl 19.1.0 pyhd3eb1b0_1 pyparsing 2.4.7 py_0 pyqt 5.9.2 py38ha925a31_4 pysocks 1.7.1 py38haa95532_0 python 3.8.2 he1778fa_13 python-dateutil 2.8.1 py_0 pytorch 1.5.0 py3.8_cuda102_cudnn7_0 pytorch pytz 2020.1 py_0 qt 5.9.7 vc14h73c81de_0 requests 2.23.0 py38_0 scipy 1.4.1 py38h9439919_0 setuptools 46.2.0 py38_0 sip 4.19.13 py38ha925a31_0 six 1.14.0 py38haa95532_0 sqlite 3.31.1 h2a8f88b_1 tk 8.6.8 hfa6e2cd_0 torchvision 0.6.0 py38_cu102 pytorch tornado 6.0.4 py38he774522_1 urllib3 1.25.8 py38_0 vc 14.2 h21ff451_1 vs2015_runtime 14.27.29016 h5e58377_2 wheel 0.34.2 py38_0 win_inet_pton 1.1.0 py38haa95532_0 wincertstore 0.2 py38_0 xz 5.2.5 h62dcd97_0 zlib 1.2.11 h62dcd97_4 zstd 1.3.7 h508b16e_0 No errors reported. Warning log Warning (roc): Error initialising ROC: No ROC toolchains found. Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init: HSA is not currently supported on this platform (win32).`
END CUDA INFO
The section below is most likely not needed for this question, here in case:
DETAILS PART 2 (It sounded like dlib (and cmake) were only required to align the faces before therefore theoretically it isn't needed in run.py, but I'm not 100 percent sure. I omitted the section including the dlib specific errors):
I got the .yml installed successfully into the anaconda environment I named "pulse3" with no errors as well as the CUDA toolkit 10.2 and cmake 3.17.2. Dlib is the only thing that gave me some trouble spiting out a bunch of incompatibility errors with the other packages on windows with my python version 3.8.2.