EMR PySpark "ModuleNotFoundError: No module named 'spacy'"

user_3013565 0
Viewed 753

I've been unsuccessfully trying to install Spacy onto my EMR cluster to run a Pyspark job. My Bootstrapping actions to EMR look something like this

pip install --upgrade pip

sudo conda install -c conda-forge spacy
sudo python3 -m spacy download en_core_web_sm
sudo python3 -m spacy download en 

sudo python3 -m pip install -U spacy
sudo python3 -m pip install -U boto3
sudo python3 -m pip install -U pandas
sudo python3 -m spacy download en_core_web_sm
sudo python3 -m spacy download en

As you see above i've been trying to install it via pip and conda both but none seem to work. Suprisingly when I use a jupyter notebook and not try to submit my pyspark job as a step to EMR it works.

amazon-emr conda pyspark spacy pip
1 Answers

I had faced a similar problem. Some of the things that could work:

  1. Check the stdout, stderr files in EMR, on bootstrap actions. It's mentioned under the summary section of the cluster-Configuration details-LOG URI
  2. Apparently, Spacy has Cython dependency and it's not downloaded automatically. Thus including the following commands helped:
    sudo python3 -m pip install --upgrade pip

    sudo python3 -m pip install --upgrade pip setuptools
    
    sudo python3 -m pip install wheel
    
    sudo python3 -m pip install -U Cython

    sudo python3 -m pip install -U spacy==2.3.5

    sudo python3 -m spacy download en_core_web_sm
user_99399780
Related