How to install boto3 onto an EMR cluster for use with Jupyter Notebook

Question

I'm having issues getting boto3 installed on EMR. Here is the bootstrap script I'm currently using:

#!/bin/bash
# Install Python 3 kernel
sudo yum install python3
sudo yum install python3-pip
sudo pip3 install -U boto3

I checked the logs and it says that boto3 was successfully installed, but when I try to import it in a Jupyter notebook I get ModuleNotFoundError: No module named 'boto3'

user_6501026 · Answer

Figured it out. Launch a notebook with the pyspark kernel and run this:

sc.install_pypi_package(“boto3”)

How to install boto3 onto an EMR cluster for use with Jupyter Notebook

1 Answers