How to enable ssl on SparkSession in python

user_1488601 0
Viewed 1601

We are using AWS Glue to connect to our Postgres DB.

from pyspark.sql import SparkSession
from pyspark.conf import SparkConf
ss = SparkSession.builder.appName("profile-dump-dev").getOrCreate()
c = SparkConf()
ss.builder.config(conf=c)
...
x = ss.read.format("jdbc").option("url",url).option("query","select * from foo").load()
# The above line throws an exception ^

The exception is:

FATAL: no pg_hba.conf entry for host "10.20.0.153", user "XXX"

In our pg_hba.conf we have (which means SSL must be enable):

hostssl X X X X

We can access the Postgres DB without issue via tools like psql and django and whatnot. When we access it the DB via normal methods, we see in the postgres logs:

connection authorized: user=XXX database=XXX SSL enabled (protocol=TLSv1.2, cipher=ECDHE-RSA-AES256-GCM-SHA384, compression=off)",,,,,,,,,"

So in a nutshell, how do we enable SSL when using SparkSession?

According to this doc: https://jdbc.postgresql.org/documentation/head/ssl-client.html We can pass this into the driver: "ssl=true".

But how do you pass ssl=true to the driver? We are using python.

aws-glue pyspark python
1 Answers

We talked to AWS Support. This is the code that makes it work:

  remote_table = spark.read.format("jdbc")\
      .option("driver", driver)\
      .option("url", url)\
      .option("dbtable", table)\
      .option("user", user)\
      .option("password", password)\
      .option("ssl", True) \
      .option("sslmode", "require" ) \
      .load()

That magic there is sslmode == require. You don't have to explicitly use any certs or anything. We don't have any custom certs at all, which is why our connection object works fine (it uses SSL under the covers), but for sparck, you explicitly need to say ssl == True and sslmode == require.

user_14886010
Related