Environment Variable Error when running Python/PySpark script

user_16994790 0
Viewed 3908

Is there an easy way to fix this error:

Missing Python executable 'python3', defaulting to 'C:\Users\user1\Anaconda3\Lib\site-packages\pyspark\bin\..' for SPARK_HOME environment variable. Please install Python or specify the correct Python executable in PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON environment variable to detect SPARK_HOME safely.

Would I have to modify the PATH system variable? Or export/create the environment variables PYSPARK_DRIVER_PYTHON and PYSPARK_PYTHON? I have Python 3.8.8.

pyspark apache-spark python
2 Answers

you need to add an environment variable called SPARK_HOME: this variable contain the path to the installed pyspark library .

In my case , pyspark is installed under my home directory, so this is the content of the variable :

SPARK_HOME=/home/zied/.local/lib/python3.8/site-packages/pyspark

also you need another variable called PYSPARK_PYTHON which have the python version you are using like this :

PYSPARK_PYTHON=python3.8
user_97715600

Just put the path in your jupyter notebook in the following code and RUN IT EVERYTIME:-

import os

os.environ["PYSPARK_PYTHON"] = r"C:\Users\LAPTOP0534\miniconda3\envs\pyspark_v3.3.0"

os.environ["PYSPARK_DRIVER_PYTHON"] = r"C:\Users\LAPTOP0534\miniconda3\envs\pyspark_v3.3.0"

os.environ["JAVA_HOME"] = r"C:\Program Files\Java\jre1.8.0_333"

I have gone through many answers but nothing works for me.
But it works in my case. This has resolved my error.

user_182654400
Related