I have created a glue ETL job, which is loading data from file present at s3 location into Redshift table. However my need is to load the data into a particular redshift schema, below is the sample code from my glue script:
S3bucket_node1 = glueContext.create_dynamic_frame.from_options(
format_options={"withHeader": True, "separator": "~"},
connection_type="s3",
format="csv",
connection_options={"paths": ["s3://<bucketName>/{0}".format(file_name_temp)]},
transformation_ctx="S3bucket_node1",
)
print('{0} completed'.format(file_name))
print(S3bucket_node1)
datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = S3bucket_node1, catalog_connection = "redshift", connection_options = {"dbtable": "Persons", "database": "dev"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink4")
With above script, it always load data into Public schema. When I try to define DBname as "Schema.tableName" it gives an error. I tried to add a new property as "Schema" in connection_options that did not worked as well. Can someone please help, how can I define the schema name above to load data into that particular schema only. I knw how it can be done by creating the crawler, but in my project we are restricted to create crawlers.