I keep trying to run pyspark and am running into a common error but none of the solutions online seem to work.
Below is the code that triggers the error
spark = SparkSession.builder \
.master("local[*]") \
.appName('test') \
.getOrCreate()
and here is the error
spark = SparkSession.builder
File "C:\tools\spark-3.0.3-bin-hadoop3.2\python\pyspark\sql\session.py", line 186, in getOrCreate
sc = SparkContext.getOrCreate(sparkConf)
File "C:\tools\spark-3.0.3-bin-hadoop3.2\python\pyspark\context.py", line 378, in getOrCreate
SparkContext(conf=conf or SparkConf())
File "C:\tools\spark-3.0.3-bin-hadoop3.2\python\pyspark\context.py", line 133, in __init__
SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
File "C:\tools\spark-3.0.3-bin-hadoop3.2\python\pyspark\context.py", line 327, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "C:\tools\spark-3.0.3-bin-hadoop3.2\python\pyspark\java_gateway.py", line 98, in launch_gateway
proc = Popen(command, **popen_kwargs)
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 947, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Users\chris\AppData\Local\Programs\Python\Python39\lib\subprocess.py", line 1416, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
FileNotFoundError: [WinError 2] The system cannot find the file specified
I've gone over my PATH variables over and over. see links to pictures below.
environment variables screenshot
environment variables screenshot cont...
I've tried every order of the variables in the path variables and the use of %variable% vs just writing out the path. the winutils.exe is in both the HADOOP_HOME\bin folder and the SPARK_HOME\bin folder.
I am able to run the spark-shell.cmd without much issue and can do things like val and what not. see picture link below
I feel like there is something glaring im missing since no posted solution appears to work.