How can I access a table in a AWS KMS encrypted redshift cluster from a glue job using pyspark script?

Viewed 129

My requirement:

I want to write a pyspark script to read data from a table in a AWS KMS encrypted redshift cluster(required SSL is true).

How can I retrieve connection details like password and use it connect to redshift like in the sample code?

What is the standard way to perform this?

Do I have to use any api?

I know that the below command generates temporary password, but this password does not work in glue redshift connection. Plus, it is not the recommended way I believe.

aws redshift get-cluster-credentials --db-user adminuser --db-name dev --cluster-identifier mycluster

My sample glue spark script:

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
from awsglue.context import GlueContext
from awsglue.job import Job

## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
#job.commit()
connection_options = {
    "url": "jdbc:redshift://endpoint",
    "dbtable": "some_table",
    "user": "user",
    "password": "some_password",  # how can I retrieve password and avoid plaintext?
    "redshiftTmpDir": args["TempDir"]
}
df = glueContext.create_dynamic_frame_from_options("redshift", connection_options).toDF()
print(df.count())
0 Answers
Related