how to provide redshift schema name in glue job

Viewed 17

I have created a glue ETL job, which is loading data from file present at s3 location into Redshift table. However my need is to load the data into a particular redshift schema, below is the sample code from my glue script:

            S3bucket_node1 = glueContext.create_dynamic_frame.from_options(
                format_options={"withHeader": True, "separator": "~"},
                connection_type="s3",
                format="csv",
                connection_options={"paths": ["s3://<bucketName>/{0}".format(file_name_temp)]},
                transformation_ctx="S3bucket_node1",
            )
            print('{0} completed'.format(file_name))
            print(S3bucket_node1)
            datasink4 = glueContext.write_dynamic_frame.from_jdbc_conf(frame = S3bucket_node1, catalog_connection = "redshift", connection_options = {"dbtable": "Persons", "database": "dev"}, redshift_tmp_dir = args["TempDir"], transformation_ctx = "datasink4")

With above script, it always load data into Public schema. When I try to define DBname as "Schema.tableName" it gives an error. I tried to add a new property as "Schema" in connection_options that did not worked as well. Can someone please help, how can I define the schema name above to load data into that particular schema only. I knw how it can be done by creating the crawler, but in my project we are restricted to create crawlers.

0 Answers
Related