Cannot get spark/pyspark working on macOS Mojave

user_12738953 0
Viewed 269

Have a new MacBook w/ macOS Mojave but cannot get spark/pyspark working. I have done the following:

  • Downloaded the latest Apache Spark from https://spark.apache.org/downloads.html
  • Unzipped it in my home directory (/Users/myname/)
  • Renamed the folder spark
  • Checked in the Finder window that spark was there, and the folder contained python, bin, and bin/spark-shell subfolders
  • Got the ability to pip install packages with: curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py >> python get-pip.py in the terminal >> Checked that it was installed by doing import pip in the python shell >> sudo easy_install pip in the regular terminal
  • pip install pyspark, which worked
  • Created both a .bashrc and .bash_profile files in the Users/myname/ path
  • I tried the following versions of the commands in these bash files:
export SPARK_HOME="/Users/myname/spark/python"
export PATH="$SPARK_HOME/bin:$PATH"

export SPARK_HOME=/Users/myname/spark/python
export PATH=$SPARK_HOME/bin:$PATH

export SPARK_HOME=/Users/myname/spark
export PATH=$SPARK_HOME/bin:$PATH

export SPARK_HOME=~/spark
export PATH=$SPARK_HOME/bin:$PATH

Different installation articles say I should just be able to do pyspark in the terminal, or spark/bin/spark-shell or just 'bin/spark-shell`, I just keep getting versions of the message

-bash: pyspark: command not found

Any advice?

pyspark apache-spark macos installation
1 Answers

If you've installed pyspark from pip, you don't need the standalone installation. You just need to fix you env vars. First, find the location of your site packages:

python -c "import site; print(site.getsitepackages())"

This is probably something like this /usr/lib/pythonX.Y/site-packages

Then, add the exports to your bash profile and source it.

echo "export SPARK_HOME=/usr/lib/pythonX.Y/site-packages/pyspark" >> ~/.bash_profile
echo "export PATH=$SPARK_HOME/bin:$PATH" >> ~/.bash_profile
source ~/.bash_profile
user_60309510
Related