My requirement:
I want to write a pyspark script to read data from a table in a AWS KMS encrypted redshift cluster(required SSL is true).
How can I retrieve connection details like password and use it connect to redshift like in the sample code?
What is the standard way to perform this?
Do I have to use any api?
I know that the below command generates temporary password, but this password does not work in glue redshift connection. Plus, it is not the recommended way I believe.
aws redshift get-cluster-credentials --db-user adminuser --db-name dev --cluster-identifier mycluster
My sample glue spark script:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from pyspark.sql import SQLContext
from awsglue.context import GlueContext
from awsglue.job import Job
## @params: [JOB_NAME]
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
#job.commit()
connection_options = {
"url": "jdbc:redshift://endpoint",
"dbtable": "some_table",
"user": "user",
"password": "some_password", # how can I retrieve password and avoid plaintext?
"redshiftTmpDir": args["TempDir"]
}
df = glueContext.create_dynamic_frame_from_options("redshift", connection_options).toDF()
print(df.count())