Dask df.to_parquet can't find pyarrow. RuntimeError: `pyarrow` not installed

user_341922 0
Viewed 191

Environment:

  • macOS Big Sur v 11.6.1
  • Python 3.7.7
  • pyarrow==5.0.0 (from pipfreeze)

From the terminal:

>>> import pyarrow
>>> pyarrow
<module 'pyarrow' from '/Users/garyb/Develop/DS/tools-pay-data-pipeline/env/lib/python3.7/site-packages/pyarrow/__init__.py'

So I confirmed that I have pyarrow installed. But when I try to write a Dask dataframe to parquet I get:

def make_parquet_file(filepath):
    parquet_path = f'{PARQUET_DIR}/{company}_{table}_{batch}.parquet'
    df.to_parquet(parquet_path, engine='pyarrow')

ModuleNotFoundError: No module named pyarrow

The exception detail:

~/Develop/DS/research-dask-parquet/env/lib/python3.7/site-packages/dask/dataframe/io/parquet/core.py in get_engine(engine)
    970     elif engine in ("pyarrow", "arrow", "pyarrow-legacy", "pyarrow-dataset"):
    971 
--> 972         pa = import_required("pyarrow", "`pyarrow` not installed")
    973         pa_version = parse_version(pa.__version__)
    974

This function works. It's a much smaller csv file just to confirm that the df.to_parquet function works:

def make_parquet_file():
    csv_file = f'{CSV_DATA_DIR}/diabetes.csv'
    parquet_file = f'{PARQUET_DIR}/diabetes.parquet'

    # Just to prove I can read the csv file
    p_df = pd.read_csv(csv_file)
    print(p_df.shape)

    d_df = dd.read_csv(csv_file)
    d_df.to_parquet(parquet_file)

Is it looking in the right place for the package? I'm stuck

dask parquet python
2 Answers

It does seem that dask and pure python are using different environments.

In the first example the path is:

~/Develop/DS/tools-pay-data-pipeline/env/lib/python3.7

In the traceback the path is:

~/Develop/DS/research-dask-parquet/env/lib/python3.7

So a quick fix is to install pyarrow in the second environment. Another fix is to install the packages on workers (this might help).

A more robust fix is to use environment files.

user_106935960

OK, problem solved - maybe. During development and experimentation I use Jupyter to test and debug. Later all the functions get moved into scripts which can then be imported into any notebook that needs them. The role of the notebook at that point in this project is demo and document, i.e. a better alternative to the command line for demo.

In this case I'm still in the experiment mode. So the problem was the Jupyter kernel. I had inadvertency recycled the name, and the virtual environment in the two projects also have the same name - "env". See a pattern here? Lazy bit me.

I deleted the kernel that was being used and created a new one with a unique name. PyArrow was then pulled from the correct virtual environment and worked as expected.

user_3419220
Related