PYCHARM Error-- java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified

user_11609306 0
Viewed 7235

I am getting the below error while running a pyspark program on PYCHARM, Error:

java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified ......

The interpreter is recognizing the python.exe file and I have added the Content root in project structure.

I got a similar issue while running the same program before in on windows command prompt and solved it using What is the right way to edit spark-env.sh before running spark-shell?

pyspark pycharm python
3 Answers

Before creating your spark session, set the following environment variables in your code:

import os
import sys
from pyspark.sql import SparkSession

os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
spark = SparkSession.builder.getOrCreate()
user_113991420

create an environment variable PYSPARK_PYTHON with value 'python'. it worked for me!

user_166705500
  1. Go to Environmental variable and within System variable set a new variable as PYSPARK_PYTHON and value as python

PYSPARK_PYTHON=python

  1. Add below codebits to your pyspark code
import os
import sys
from pyspark import SparkContext
os.environ['PYSPARK_PYTHON'] = sys.executable
os.environ['PYSPARK_DRIVER_PYTHON'] = sys.executable
user_107955960
Related