python3 -q -X faulthandler test.py --model.backbone=resnet18
2021-02-07 20:14:21.198583: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
WARNING:root:Argument whitelist is deprecated. Please use allowlist.
2021-02-07 20:14:22.474523: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-07 20:14:22.474598: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2021-02-07 20:14:22.474690: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.475028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-07 20:14:22.475044: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.476599: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-07 20:14:22.476630: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-07 20:14:22.477189: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-07 20:14:22.477333: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-07 20:14:22.478386: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-07 20:14:22.478803: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-07 20:14:22.478884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-07 20:14:22.478950: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.479304: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.479601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-07 20:14:22.479771: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-02-07 20:14:22.480090: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-02-07 20:14:22.480170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.480489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.755GHz coreCount: 46 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2021-02-07 20:14:22.480501: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.480512: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2021-02-07 20:14:22.480521: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2021-02-07 20:14:22.480529: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2021-02-07 20:14:22.480538: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2021-02-07 20:14:22.480547: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2021-02-07 20:14:22.480555: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2021-02-07 20:14:22.480564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2021-02-07 20:14:22.480597: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.480916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.481206: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2021-02-07 20:14:22.481224: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2021-02-07 20:14:22.815420: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-02-07 20:14:22.815463: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267] 0
2021-02-07 20:14:22.815469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0: N
2021-02-07 20:14:22.815656: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816027: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816362: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-02-07 20:14:22.816685: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5617 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:01:00.0, compute capability: 8.6)
Fatal Python error: Segmentation fault
Current thread 0x00007f52327fa740 (most recent call first):
File "/home/oem/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172 in _lazy_init
File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 610 in convert
File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 381 in _apply
File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 359 in _apply
File "/home/oem/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 612 in to
File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 32 in get_model
File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 70 in get_multinet_extractor
File "/home/oem/sfuhome/CMPT415/SUR-master/models/model_helpers.py", line 61 in get_domain_extractors
File "test.py", line 39 in main
File "test.py", line 84 in <module>
Segmentation fault (core dumped)
My guess for this error would be related to CUDA error.
I check my CUDA version
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
But I was installed the most recent CUDA library
/usr/local$ ls
bin cuda cuda-11.2 etc games include lib man sbin share src
But nvidia-smi gives me another version
nvidia-smi
/usr/bin/nvidia-modprobe: unrecognized option: "-s"
ERROR: Invalid commandline, please run `/usr/bin/nvidia-modprobe --help` for usage information.
/usr/bin/nvidia-modprobe: unrecognized option: "-s"
ERROR: Invalid commandline, please run `/usr/bin/nvidia-modprobe --help` for usage information.
Sun Feb 7 20:19:40 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03 Driver Version: 460.32.03 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3070 Off | 00000000:01:00.0 On | N/A |
| 0% 51C P5 23W / 220W | 1628MiB / 7979MiB | 29% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1024 G /usr/lib/xorg/Xorg 768MiB |
| 0 N/A N/A 1371 G /usr/bin/gnome-shell 456MiB |
| 0 N/A N/A 1857 G ...gAAAAAAAAA --shared-files 336MiB |
+-----------------------------------------------------------------------------+
And this gives me
cat /usr/local/cuda/version.txt
cat: /usr/local/cuda/version.txt: No such file or directory
Is the problem related to the pytorch or tensorflow because I see this error comes from python3.8/site-package/torch
Anyone can help me? Thank you so much!
And this is the library version
absl-py==0.11.0
apt-clone==0.2.1
apturl==0.5.2
astroid==2.4.2
astunparse==1.6.3
attrs==20.3.0
backcall==0.1.0
bcrypt==3.1.7
bleach==3.1.1
blinker==1.4
Brlapi==0.7.0
cachetools==4.2.1
certifi==2019.11.28
cffi==1.14.4
chardet==3.0.4
Click==7.0
cloudpickle==1.6.0
colorama==0.4.3
command-not-found==0.3
cryptography==2.8
cupshelpers==1.0
cycler==0.10.0
Cython==0.29.21
dataclasses==0.6
dbus-python==1.2.16
decorator==4.4.2
defer==1.0.6
defusedxml==0.6.0
dill==0.3.3
distro==1.4.0
distro-info===0.23ubuntu1
dm-tree==0.1.5
duplicity==0.8.12.0
entrypoints==0.3
fasteners==0.14.1
flatbuffers==1.12
future==0.18.2
gast==0.3.3
gin-config==0.4.0
google-api-core==1.25.1
google-api-python-client==1.12.8
google-auth==1.24.0
google-auth-httplib2==0.0.4
google-auth-oauthlib==0.4.2
google-cloud-bigquery==2.7.0
google-cloud-core==1.5.0
google-crc32c==1.1.2
google-pasta==0.2.0
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
grpcio==1.32.0
h5py==2.10.0
html5lib==1.0.1
httplib2==0.18.1
idna==2.8
importlib-metadata==1.5.0
importlib-resources==5.1.0
ipykernel==5.2.0
ipython==7.13.0
ipython-genutils==0.2.0
ipywidgets==6.0.0
isort==5.7.0
jedi==0.15.2
Jinja2==2.10.1
joblib==1.0.0
jsonschema==3.2.0
jupyter-client==6.1.2
jupyter-core==4.6.3
kaggle==1.5.10
Keras-Preprocessing==1.1.2
keyring==18.0.1
kiwisolver==1.3.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
lazy-object-proxy==1.4.3
linecache2==1.0.0
lmdb==1.1.1
lockfile==0.12.2
louis==3.12.0
macaroonbakery==1.3.1
Mako==1.1.0
Markdown==3.3.3
MarkupSafe==1.1.0
matplotlib==3.3.4
mccabe==0.6.1
meta-dataset==0.2.0
mistune==0.8.4
monotonic==1.5
more-itertools==4.2.0
nbconvert==5.6.1
nbformat==5.0.4
netifaces==0.10.4
notebook==6.0.3
numpy==1.19.5
oauth2client==4.1.3
oauthlib==3.1.0
olefile==0.46
opencv-python-headless==4.5.1.48
opt-einsum==3.3.0
PAM==0.4.2
pandas==1.2.1
pandocfilters==1.4.2
paramiko==2.6.0
parso==0.5.2
pbr==5.4.5
pexpect==4.6.0
pickleshare==0.7.5
Pillow==7.0.0
prometheus-client==0.7.1
promise==2.3
prompt-toolkit==2.0.10
proto-plus==1.13.0
protobuf==3.14.0
psutil==5.8.0
py-cpuinfo==7.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycairo==1.16.2
pycocotools==2.0.2
pycparser==2.20
pycups==1.9.73
Pygments==2.3.1
PyGObject==3.36.0
PyICU==2.4.2
PyJWT==1.7.1
pylint==2.6.0
pymacaroons==0.13.0
PyNaCl==1.3.0
pyparsing==2.4.7
pyRFC3339==1.1
pyrsistent==0.15.5
python-apt==2.0.0+ubuntu0.20.4.4
python-dateutil==2.7.3
python-debian===0.1.36ubuntu1
python-slugify==4.0.1
pytz==2019.3
pyxdg==0.26
PyYAML==5.3.1
pyzmq==18.1.1
reportlab==3.5.34
requests==2.22.0
requests-oauthlib==1.3.0
requests-unixsocket==0.2.0
rsa==4.7
scikit-learn==0.24.1
scipy==1.6.0
screen-resolution-extra==0.0.0
SecretStorage==2.3.1
Send2Trash==1.5.0
sentencepiece==0.1.95
seqeval==1.2.2
simplejson==3.16.0
six==1.15.0
sklearn==0.0
systemd-python==234
tabulate==0.8.7
tb-nightly==2.5.0a20210128
tensorboard==2.4.1
tensorboard-plugin-wit==1.8.0
tensorflow==2.4.1
tensorflow-addons==0.12.0
tensorflow-datasets==4.2.0
tensorflow-estimator==2.4.0
tensorflow-gpu==2.4.1
tensorflow-hub==0.11.0
tensorflow-metadata==0.27.0
tensorflow-model-optimization==0.5.0
tensorflow-probability==0.7.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.4
testresources==2.0.0
text-unidecode==1.3
tf-estimator-nightly==2.5.0.dev2021012801
tf-models-official==2.4.0
tf-nightly-gpu==2.5.0.dev20210128
tf-slim==1.1.0
threadpoolctl==2.1.0
toml==0.10.2
torch==1.7.1+cu110
torchaudio==0.7.2
torchvision==0.8.2+cu110
tornado==5.1.1
tqdm==4.56.0
traceback2==1.4.0
traitlets==4.3.3
typeguard==2.10.0
typing-extensions==3.7.4.3
ubuntu-advantage-tools==20.3
ubuntu-drivers-common==0.0.0
ufw==0.36
unattended-upgrades==0.1
unittest2==1.1.0
uritemplate==3.0.1
urllib3==1.25.8
usb-creator==0.3.7
wadllib==1.3.3
wcwidth==0.1.8
webencodings==0.5.1
Werkzeug==1.0.1
widgetsnbextension==2.0.0
wrapt==1.12.1
xkit==0.0.0
zipp==1.0.0
I try three different versions of Pytorch cuda10.1,cuda11.0. None of them works
And which library is most related to this line?
File "/home/oem/.local/lib/python3.8/site-packages/torch/cuda/__init__.py", line 172 in _lazy_init