Cannot open pickle file generated by snakemake pipeline

user_4746549 0
Viewed 84

I have a data analysis pipeline that consists of multiple steps. I have generated a snakemake pipeline (new for me) and the output of every task (and input of the next task) is a pickle file containing either a DataFrame or a list of DataFrames. Everything is fine except I cannot open the pickle files manually. Of note, the pipeline uses a dedicated conda environment.

import _pickle
with open("testb/first/out/stacks.pkl", "rb") as f:
    data = _pickle.load(f)

I get this error:

AttributeError: Can't get attribute '_unpickle_block' on <module 'pandas._libs.internals' from 'C:\\Users\\sebde\\anaconda3\\envs\\dbm\\lib\\site-packages\\pandas\\_libs\\internals.cp39-win_amd64.pyd'

Python 3.10.2, Snakemake-minimal 7.0.4 (as per documentation, I'm on Windows), Pandas 1.4.1

snakemake pandas python
1 Answers

One easy thing to try (but which might not work) is to use pickle rather than _pickle:

# the "as" part is to avoid adjust downstream code
# but if this is not a concern a regular import is better
# (i.e. "import pickle")
import pickle as _pickle
user_106935960
Related