Running pyspark after pip install pyspark

user_1473517 0
Viewed 48372

I wanted to install pyspark on my home machine. I did

pip install pyspark
pip install jupyter

Both seemed to work well.

But when I try to run pyspark I get 

pyspark
Could not find valid SPARK_HOME while searching ['/home/user', '/home/user/.local/bin']

What should SPARK_HOME be set to?

pyspark pip
5 Answers

To install Spark, make sure you have Java 8 or higher installed. Then go to Spark Downloads page to select latest spark release, prebuilt package for Hadoop and download it. Unzip the file and move to your /opt (or for that matter any folder, but remember where you moved it)

mv spark-2.4.4-bin-hadoop2.7 /opt/spark-2.4.4

Then create a symbolic link. This way you can download and use multiple spark versions.

ln -s /opt/spark-2.4.4 /opt/spark

Add the following to your, .bash_profile to tell your bash where to find Spark.

export SPARK_HOME=/opt/spark
export PATH=$SPARK_HOME/bin:$PATH

Finally, to setup Spark to use python3, please add the following to /opt/spark/conf/spark-env.sh file

export PYSPARK_PYTHON=/usr/local/bin/python3
export PYSPARK_DRIVER_PYTHON=python3
user_118048130

If you are in python 3.0+ then open anaconda prompt execute the below command pip3 install --user pyspark

user_93079220

Easiest way - Open anaconda prompt and type pip install --user pyspark / pip3 install --user pyspark

user_122230410
Related