I've been trying to setup PYSPARK_PYTHON from a juputer notebook(using jupyter lab) to use a specific conda env but i cannot find a way to make it work, I have found some examples using:
import os
os.environ['PYSPARK_PYTHON'] = "<the path>"
But it did not work so I also tried:
spark = pyspark.sql.SparkSession.builder \
.master("yarn-client") \
.appName(session_name) \
.config("spark.yarn.appMasterEnv.PYSPARK_PYTHON","<the path>") \
.enableHiveSupport() \
.getOrCreate(cluster=cluster)
sc = spark.sparkContext
sqlContext = SQLContext(sc)
But it never uses the specified python version in the specified path , question is, is it possible the config is being ignored? do something else needs to be done in notebook?
I'm using yarn-client mode, and i'm using an enterprise/corporate instance of jupyter lab so i cannot set the variables on the cli using export because the server is widely managed for the company by another team so i need to do it with something different than export at cli during jupyter start.