How do i save a single value output of a Spark.SQL output as a variable to use further in code

user_17299405 0
Viewed 61

I tried using the below option, but it is saving the value as a data frame, but I only need the value as a variable for processing

value = spark.read.format("net.snowflake.spark.snowflake").options(**sfOptions).option("query", SQL).load()
snowflake-cloud-data-platform pyspark python
1 Answers

To store the value to a variable, you need an actual action. The .load you have in there is just a pointer to the underlying data. It doesn't materialize your query.

One way is to create a temp view and perform a first or collect action after spark.sql() call.

spark.read \
     .format("net.snowflake.spark.snowflake") \
     .options(**sfOptions) \ 
     .option("query", SQL) \ --feel free to remove this 
     .load() \
     .createOrReplaceTempView("my_data")

my_var = spark.sql("select that single value from my_data").first()[0]
user_99471590
Related