Currently using Python = 3.5 and Spark = 2.4 versions.
I am trying to run PySpark in a Linux context (git bash) on a Windows machine and I get the following:
set PYSPARK_SUBMIT_ARGS="--name" "PySparkShell" "pyspark-shell" && python3
Does anyone know where I should set these variables? I couldnt't find anything that works for me on google.
spark-shell with Scala works, so I am guessing is something related to the Python config.