I created a minimal example with this structure
./Snakefile
./simple_file.py
./envs/binvestigate_3_6_fresh.yml
Snakefile contains
rule step_1:
input:
output:
"step_1_dummy.txt"
conda:
"envs/binvestigate_3_6_fresh.yml"
script:
"simple_file.py"
simple_file.py contains
import pandas
import os
if __name__=="__main__":
os.system('touch /home/rictuar/coding_projects/delete_crap/step_1_dummy.txt')
binvestigate_3_6_fresh.yml contains
name: binvestigate_3_6_fresh
channels:
- conda-forge
- bioconda
- defaults
dependencies:
- _libgcc_mutex=0.1=conda_forge
- _openmp_mutex=4.5=1_gnu
- aioeasywebdav=2.4.0=py36h5fab9bb_1001
- aiohttp=3.7.4.post0=py36h8f6f2f9_0
- alsa-lib=1.2.3=h516909a_0
- appdirs=1.4.4=pyh9f0ad1d_0
- async-timeout=3.0.0=py36_0
- attrs=21.2.0=pyhd8ed1ab_0
- ca-certificates=2021.10.8=ha878542_0
- certifi=2016.9.26=py36_0
- chardet=4.0.0=py36h5fab9bb_1
- configargparse=1.5.3=pyhd8ed1ab_0
- dbus=1.13.6=h48d8840_2
- docutils=0.16=py36h5fab9bb_3
- dropbox=7.3.1=py36_0
- ecdsa=0.17.0=pyhd8ed1ab_0
- ete3=3.1.2=pyh9f0ad1d_0
- expat=2.4.1=h9c3ff4c_0
- filechunkio=1.8=py_2
- fontconfig=2.13.1=hba837de_1005
- freetype=2.10.4=h0708190_1
- ftputil=5.0.1=pyhd8ed1ab_0
- gettext=0.19.8.1=h73d1719_1008
- glib=2.70.0=h780b84a_1
- glib-tools=2.70.0=h780b84a_1
- gst-plugins-base=1.18.5=hf529b03_2
- gstreamer=1.18.5=h9f60fe5_2
- icu=68.2=h9c3ff4c_0
- idna=3.1=pyhd3deb0d_0
- idna_ssl=1.1.0=pyhd8ed1ab_1002
- jpeg=9d=h36c2ea0_0
- krb5=1.19.2=hcc1bbae_3
- ld_impl_linux-64=2.36.1=hea4e1c9_2
- libblas=3.9.0=12_linux64_openblas
- libcblas=3.9.0=12_linux64_openblas
- libclang=11.1.0=default_ha53f305_1
- libedit=3.1.20191231=he28a2e2_2
- libevent=2.1.10=h9b69904_4
- libffi=3.4.2=h7f98852_5
- libgcc-ng=11.2.0=h1d223b6_11
- libgfortran-ng=11.2.0=h69a702a_11
- libgfortran5=11.2.0=h5c6108e_11
- libglib=2.70.0=h174f98d_1
- libgomp=11.2.0=h1d223b6_11
- libiconv=1.16=h516909a_0
- liblapack=3.9.0=12_linux64_openblas
- libllvm11=11.1.0=hf817b99_2
- libogg=1.3.4=h7f98852_1
- libopenblas=0.3.18=pthreads_h8fe5266_0
- libopus=1.3.1=h7f98852_1
- libpng=1.6.37=h21135ba_2
- libpq=13.3=hd57d9b9_3
- libstdcxx-ng=11.2.0=he4da1e4_11
- libuuid=2.32.1=h7f98852_1000
- libvorbis=1.3.7=h9c3ff4c_0
- libxcb=1.13=h7f98852_1004
- libxkbcommon=1.0.3=he3ba5ed_0
- libxml2=2.9.12=h72842e0_0
- libxslt=1.1.33=h15afd5d_2
- libzlib=1.2.11=h36c2ea0_1013
- lxml=3.8.0=py36_0
- lz4-c=1.9.3=h9c3ff4c_1
- multidict=5.2.0=py36h8f6f2f9_0
- mysql-common=8.0.27=ha770c72_1
- mysql-libs=8.0.27=hfa10184_1
- ncurses=6.2=h58526e2_4
- nspr=4.32=h9c3ff4c_1
- nss=3.72=hb5efdd6_0
- numpy=1.19.5=py36hfc0c790_2
- openssl=1.1.1l=h7f98852_0
- pandas=1.1.5=py36h284efc9_0
- paramiko=1.18.2=py36_0
- pcre=8.45=h9c3ff4c_0
- pip=21.3.1=pyhd8ed1ab_0
- psutil=5.8.0=py36h8f6f2f9_1
- pthread-stubs=0.4=h36c2ea0_1001
- pycrypto=2.6.1=py36he6145b8_1005
- pyqt=5.12.3=py36h5fab9bb_7
- pyqt-impl=5.12.3=py36h7ec31b9_7
- pyqt5-sip=4.19.18=py36hc4f0c31_7
- pyqtchart=5.12=py36h7ec31b9_7
- pyqtwebengine=5.12.1=py36h7ec31b9_7
- pysftp=0.2.9=py_1
- python=3.6.13=hb7a2778_2_cpython
- python-dateutil=2.8.2=pyhd8ed1ab_0
- python_abi=3.6=2_cp36m
- pytz=2021.3=pyhd8ed1ab_0
- pyyaml=5.4.1=py36h8f6f2f9_1
- qt=5.12.9=hda022c4_4
- ratelimiter=1.2.0=py_1002
- readline=8.1=h46c0cb4_0
- requests=2.12.5=py36_0
- scipy=1.5.3=py36h81d768a_1
- setuptools=49.6.0=py36h5fab9bb_3
- six=1.16.0=pyh6c4a22f_0
- snakemake=4.3.0=py36_0
- sqlite=3.36.0=h9cd32fc_2
- tk=8.6.11=h27826a3_1
- typing-extensions=3.10.0.2=hd8ed1ab_0
- typing_extensions=3.10.0.2=pyha770c72_0
- urllib3=1.12=py36_0
- wheel=0.37.0=pyhd8ed1ab_1
- wrapt=1.12.1=py36h8f6f2f9_3
- xorg-kbproto=1.0.7=h7f98852_1002
- xorg-libice=1.0.10=h7f98852_0
- xorg-libsm=1.2.3=hd9c2040_1000
- xorg-libx11=1.7.2=h7f98852_0
- xorg-libxau=1.0.9=h7f98852_0
- xorg-libxdmcp=1.1.3=h7f98852_0
- xorg-libxext=1.3.4=h7f98852_1
- xorg-libxrender=0.9.10=h7f98852_1003
- xorg-renderproto=0.11.1=h7f98852_1002
- xorg-xextproto=7.3.0=h7f98852_1002
- xorg-xproto=7.0.31=h7f98852_1007
- xz=5.2.5=h516909a_1
- yaml=0.2.5=h516909a_0
- yarl=1.6.3=py36h8f6f2f9_2
- zlib=1.2.11=h36c2ea0_1013
- zstd=1.5.0=ha95c52a_0
prefix: /home/rictuar/anaconda3/envs/binvestigate_3_6_fresh
When I run
snakemake "/home/rictuar/coding_projects/delete_crap/step_1_dummy.txt" --core 5 --use-conda
I get
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 5
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 step_1
1
Select jobs to execute...
[Mon Nov 15 15:01:20 2021]
rule step_1:
output: /home/rictuar/coding_projects/delete_crap/step_1_dummy.txt
jobid: 0
Activating conda environment: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
Activating conda environment: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
Traceback (most recent call last):
File "/home/rictuar/coding_projects/delete_crap/.snakemake/scripts/tmpicplrg37.simple_file.py", line 3, in <module>
import sys; sys.path.extend(['/home/rictuar/anaconda3/lib/python3.8/site-packages', '/home/rictuar/coding_projects/delete_crap']); import pickle; snakemake = pickle.loads(b'\x80\x04\x95\xb4\x03\x00\x00\x00\x00\x00\x00\x8c\x10snakemake.script\x94\x8c\tSnakemake\x94\x93\x94)\x81\x94}\x94(\x8c\x05input\x94\x8c\x0csnakemake.io\x94\x8c\nInputFiles\x94\x93\x94)\x81\x94}\x94(\x8c\x06_names\x94}\x94\x8c\x12_allowed_overrides\x94]\x94(\x8c\x05index\x94\x8c\x04sort\x94eh\x0f\x8c\tfunctools\x94\x8c\x07partial\x94\x93\x94h\x06\x8c\x19Namedlist._used_attribute\x94\x93\x94\x85\x94R\x94(h\x15)}\x94\x8c\x05_name\x94h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bub\x8c\x06output\x94h\x06\x8c\x0bOutputFiles\x94\x93\x94)\x81\x94\x8c:/home/rictuar/coding_projects/delete_crap/step_1_dummy.txt\x94a}\x94(h\x0b}\x94h\r]\x94(h\x0fh\x10eh\x0fh\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bub\x8c\x06params\x94h\x06\x8c\x06Params\x94\x93\x94)\x81\x94}\x94(h\x0b}\x94h\r]\x94(h\x0fh\x10eh\x0fh\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bub\x8c\twildcards\x94h\x06\x8c\tWildcards\x94\x93\x94)\x81\x94}\x94(h\x0b}\x94h\r]\x94(h\x0fh\x10eh\x0fh\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bub\x8c\x07threads\x94K\x01\x8c\tresources\x94h\x06\x8c\tResources\x94\x93\x94)\x81\x94(K\x01K\x01e}\x94(h\x0b}\x94(\x8c\x06_cores\x94K\x00N\x86\x94\x8c\x06_nodes\x94K\x01N\x86\x94uh\r]\x94(h\x0fh\x10eh\x0fh\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bhTK\x01hVK\x01ub\x8c\x03log\x94h\x06\x8c\x03Log\x94\x93\x94)\x81\x94}\x94(h\x0b}\x94h\r]\x94(h\x0fh\x10eh\x0fh\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x0fsNt\x94bh\x10h\x13h\x15\x85\x94R\x94(h\x15)}\x94h\x19h\x10sNt\x94bub\x8c\x06config\x94}\x94\x8c\x04rule\x94\x8c\x06step_1\x94\x8c\x0fbench_iteration\x94N\x8c\tscriptdir\x94\x8c)/home/rictuar/coding_projects/delete_crap\x94ub.'); from snakemake.logging import logger; logger.printshellcmds = False; __real_file__ = __file__; __file__ = '/home/rictuar/coding_projects/delete_crap/simple_file.py';
AttributeError: Can't get attribute 'Namedlist._used_attribute' on <module 'snakemake.io' from '/home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e/lib/python3.6/site-packages/snakemake/io.py'>
[Mon Nov 15 15:01:23 2021]
Error in rule step_1:
jobid: 0
output: /home/rictuar/coding_projects/delete_crap/step_1_dummy.txt
conda-env: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
RuleException:
CalledProcessError in line 11 of /home/rictuar/coding_projects/delete_crap/Snakefile:
Command 'source /home/rictuar/anaconda3/bin/activate '/home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e'; set -euo pipefail; python /home/rictuar/coding_projects/delete_crap/.snakemake/scripts/tmpicplrg37.simple_file.py' returned non-zero exit status 1.
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
File "/home/rictuar/coding_projects/delete_crap/Snakefile", line 11, in __rule_step_1
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/home/rictuar/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/rictuar/coding_projects/delete_crap/.snakemake/log/2021-11-15T150120.368767.snakemake.log
Expected behavior is that this would simply work. When I
conda activate binvestigate_fresh_3_6
This code runs without issue. But, snakemake should be able to make conda environments on the fly. Intrestingly, it seems like snakemake is trying to use whatever environment I am in when i run Snakemake. Notice that it searches for python 3.8 when I run it from (base)
The logfile
Building DAG of jobs...
Using shell: /bin/bash
Provided cores: 5
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 step_1
1
Select jobs to execute...
[Mon Nov 15 15:01:20 2021]
rule step_1:
output: /home/rictuar/coding_projects/delete_crap/step_1_dummy.txt
jobid: 0
Activating conda environment: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
Activating conda environment: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
[Mon Nov 15 15:01:23 2021]
Error in rule step_1:
jobid: 0
output: /home/rictuar/coding_projects/delete_crap/step_1_dummy.txt
conda-env: /home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e
RuleException:
CalledProcessError in line 11 of /home/rictuar/coding_projects/delete_crap/Snakefile:
Command 'source /home/rictuar/anaconda3/bin/activate '/home/rictuar/coding_projects/delete_crap/.snakemake/conda/b84153cbaa4647564a32063b6a2b831e'; set -euo pipefail; python /home/rictuar/coding_projects/delete_crap/.snakemake/scripts/tmpicplrg37.simple_file.py' returned non-zero exit status 1.
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
File "/home/rictuar/coding_projects/delete_crap/Snakefile", line 11, in __rule_step_1
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 568, in _callback
File "/home/rictuar/anaconda3/lib/python3.8/concurrent/futures/thread.py", line 57, in run
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
File "/home/rictuar/anaconda3/lib/python3.8/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /home/rictuar/coding_projects/delete_crap/.snakemake/log/2021-11-15T150120.368767.snakemake.log
e