How do I setup pyspark in VS Code?

Question

I'm trying to run pypsark in VS-Code and I can't seem to point my environment to the correct pyspark driver and path. When I run pyspark in my terminal window it looks like this:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 3.0.1
      /_/

Using Python version 3.8.3 (default, Jul  2 2020 11:26:31)
SparkSession available as 'spark'.

When I run a file in in Vs-Code I get this error:

File "/Users/myuser/opt/anaconda3/lib/python3.8/site-packages/pyspark/python/lib/pyspark.zip/pyspark/worker.py", line 477, in main
    ("%d.%d" % sys.version_info[:2], version))
Exception: Python in worker has different version 2.7 than that in driver 3.8, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

In my bash_profile I think I have the setup correct:

export SPARK_PATH=~/spark-1.6.0-bin-hadoop2.6
export PYSPARK_PYTHON=python3
alias snotebook='$SPARK_PATH/bin/pyspark --master local[2]'

I get that it's looking at two different versions so, what do I need to do to configure my environment correctly in VS code to run pyspark?

user_17437819 · Answer

I had the same problem. It was solved by adding the following lines to /conf/spark-env.sh:

export PYSPARK_PYTHON=python3.8
export PYSPARK_DRIVER_PYTHON=python3.8

AND in vscode setting python interpreter to 3.8 too (you can set it from command palette and typing Python:Select Interpreter.

If you don't have spark-env.sh, make a copy of the template file already in the conf folder.

How do I setup pyspark in VS Code?

1 Answers