Connect to Snowflake using Databricks Snowflake connector via Okta authentication

user_2784286 0
Viewed 1117

I am trying to connect to Snowflake from Databricks using Spark connector as mentioned here. In the examples, the connection is established using the user name and password of Snowflake account. However, in my case, I am authenticating via Okta. I can see there is an option available of Okta authentication to connect using Python connector. However, I could not find the same for Spark dataframe.

Also for production pipeline deployment, is it recommended to have Okta authentication?

Has anyone ever had a similar use-case or are aware about the setup? Your help will be highly appreciated.

Thanks, Surya

snowflake-cloud-data-platform pyspark databricks apache-spark python
1 Answers

Snowflake's Spark Connector uses the JDBC driver to establish a connection to Snowflake, so the connectivity parameters of Snowflake's apply in the Spark connector as well.

The JDBC driver has the "authenticator=externalbrowser" parameter to enable SSO/Federated authentication. You can also set this parameter to your Okta endpoint for native Okta authentication.

The below code snippet demonstrate how to add this setting in a simple PySpark program that leverages the Snowflake Spark Connector. Please note that you do need to set the user and password as these are required parameters. You may set these to dummy values should you wish.

from pyspark import SparkConf, SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark import SparkConf, SparkContext
 
sc = SparkContext("local", "Simple App")
spark = SQLContext(sc)
spark_conf = SparkConf().setMaster('local').setAppName('repro')
 
sfOptions = {
 "sfURL" : "accountname.eu-central-1.snowflakecomputing.com",
 "sfAccount" : "accountname",
 "sfUser" : "manuel",
 "authenticator" : "externalbrowser",
 "sfPassword" : "xxx",
 "sfDatabase" : "SANDBOX",
 "sfSchema" : "PUBLIC",
 "sfWarehouse" : "MANUEL_WH",
 "tracing" : "ALL",
}
 
SNOWFLAKE_SOURCE_NAME = "net.snowflake.spark.snowflake"
 
df = spark.read.format(SNOWFLAKE_SOURCE_NAME) \
   .options(**sfOptions) \
   .option("query",  "select 1 as my_num union all select 2 as my_num") \
   .load()
user_30633810
Related