Installing pyarrow: can't copy 'build/lib.macosx-11-arm64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file

user_15629542 0
Viewed 431

I am trying to install pyarrow with the following command*:

OPENSSL_ROOT_DIR=/opt/homebrew/opt/openssl@1.1/ pip install "pyarrow==4.0.1" --no-use-pep517

However, it looks like the compilation fails as I get the following message at the end:

    Moving built C-extension release/_compute.cpython-39-darwin.so to build path /private/var/folders/kw/dnbqlh8n5zb9b1529n95j91m0000gr/T/pip-install-9qbgfsx2/pyarrow_3accc74c8d1a4153894730bae7812439/build/lib.macosx-11-arm64-3.9/pyarrow/_compute.cpython-39-darwin.so
    Did not find release/_cuda.cpython-39-darwin.so
    Cython module _cuda failure permitted
    Did not find release/_flight.cpython-39-darwin.so
    Cython module _flight failure permitted
    Did not find release/_dataset.cpython-39-darwin.so
    Cython module _dataset failure permitted
    Did not find release/_parquet.cpython-39-darwin.so
    Cython module _parquet failure permitted
    Did not find release/_orc.cpython-39-darwin.so
    Cython module _orc failure permitted
    Did not find release/_plasma.cpython-39-darwin.so
    Cython module _plasma failure permitted
    Did not find release/_s3fs.cpython-39-darwin.so
    Cython module _s3fs failure permitted
    Did not find release/_hdfs.cpython-39-darwin.so
    Cython module _hdfs failure permitted
    Did not find release/gandiva.cpython-39-darwin.so
    Cython module gandiva failure permitted
    running install_lib
    copying build/lib.macosx-11-arm64-3.9/pyarrow/_generated_version.py -> [*my python path*]/lib/python3.9/site-packages/pyarrow
    error: can't copy 'build/lib.macosx-11-arm64-3.9/pyarrow/include/arrow': doesn't exist or not a regular file

What is the problem here? Why does the file not exist, and how do I fix it?

*Explanation of the command:

  • OPENSSL_ROOT_DIR is required otherwise CMake complains about not finding OpenSSL
  • 4.0.1 is the most recent version and I get the same error for older versions as well
  • no-use-pep517 because apparently the build involves also building a wheel for numpy, which doesn't work with pep517 on M1 macs
apple-m1 python-3.9 pyarrow pip cmake
1 Answers

UPDATE

You can use the nightly install of pyarrow which now supports M1

pip install --extra-index-url https://pypi.fury.io/arrow-nightlies/ --prefer-binary --pre pyarrow

PRE-UPDATE

The bad news is I think this is the fault of pyarrow rather than yourself.

The good news is that I think it is about to be fixed!

If you download the wheel mentioned in this comment and then do pip install ~/Downloads/pyarrow-5.0.0.dev471-cp39-cp39-macosx_11_0_arm64.whl I think it will install.

Hopefully this will make its way into a proper release very soon.

user_59947820
Related