I recently started using pyspark on an AWS EMR cluster to do some analytics. I want to install and use some packages for this, mostly for plotting. Some of the installs happen in this bootstrap script
#!/bin/sh
set -e -x
# boto 3
pip3 install boto3 --user
# pandas
pip3 install Cython --user
pip3 install numpy==1.21.1 --user
pip3 install pandas==1.3.0 --user
# plotnine
pip3 install plotnine --user
# matplotlib
pip3 install plotnine --user
# delta table jar
spark-shell --packages io.delta:delta-core_2.11:0.3.0
The packages do not get installed for either python3 or pyspark. I can however go into a python3 console and pip install. They will then work with python3, but not pyspark, and installing them on a pyspark console does not work either.
My last ditch effort was using sc.install_pypi_package(...) on pyspark but this won't work with all the packages.
For context here is some info on the EMR
- release label: emr-6.3.1
- Hadoop dist: Amazon 3.2.1
- Applications: spark 3.1.1, livy 0.7.0, Hive 3.1.2, JupyterEnterpirceGateway 2.1.0
- Running on 1 c4.4xl instance as the master node