How to install and use pyspark on mac

wram19 0
Viewed 21

I'm taking a machine learning course and am trying to install pyspark to complete some of the class assignments. I downloaded pyspark from this link, unzipped it and put it in my home directory, and added the following lines to my .bash_profile.

export SPARK_PATH=~/spark-3.3.0-bin-hadoop2.6 
export PYSPARK_DRIVER_PYTHON="jupyter" 
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"

However, when I try to run the command:

pyspark

to start a session, I get the error:

-bash: pyspark: command not found

Can someone tell me what I need to do to get pyspark working on my local machine? Thank you.

installation pyspark
1 Answers

You are probably missing the PATH entry. Here are the environment variable changes I did to get pyspark working on my Mac:

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk-11.0.6.jdk/Contents/Home/
export SPARK_HOME=/opt/spark-3.3.0-bin-hadoop3
export PATH=$JAVA_HOME/bin:$SPARK_HOME:$SPARK_HOME/bin:$SPARK_HOME/sbin:$PATH
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON='jupyter'
export PYSPARK_DRIVER_PYTHON_OPTS='notebook --no-browser --port=8889'

Also ensure that, you've Java SE 8+ and Python 3.5+ installed.

Start the server from /opt/spark-3.3.0-bin-hadoop3/sbin/start-master.sh. Then run pyspark and copy+paste URL displayed on screen in web browser.

Azhar Khan0
Related