Can't get pip install to work on EMR cluster

user_237696 0
Viewed 2948

I have an EMR (emr-5.30.0) cluster I'm trying to start with a bootstrap file in S3. The contents of the bootstrap file are: 

#!/bin/bash
sudo pip3 install --user \
     matplotlib \
     pandas \
     pyarrow \
     pyspark

And the error in my stderr file is: 

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.
Command "python setup.py egg_info" failed with error code 1 in /mnt/tmp/pip-build-br9bn1h3/pyspark/

Seems pretty simple...no idea what is going on. Any help is appreciated.

EDIT:

Tried @Dennis Traub suggestion and get same error. New EMR bootstrap looks like this: 

#!/bin/bash
sudo pip3 install --upgrade setuptools
sudo pip3 install --user matplotlib pandas pyarrow pyspark
amazon-emr pyspark apache-spark amazon-web-services
2 Answers
#!/bin/bash

sudo python3 -m pip install matplotlib pandas pyarrow

DO NOT install pyspark. It should be already there in EMR with required config. Installing may cause problems.

user_78577010

You might have an outdated version of setuptools. Try the following script:

#!/bin/bash
sudo pip3 install --upgrade setuptools
sudo pip3 install --user matplotlib pandas pyarrow pyspark
user_1586680
Related