I have written a glue code, which reads data from S3 bucket and load into Redshift. This code is generic, so that it can read all the files from S3 in loop and load all of them in the redshift.
However in some files, I have numeric fields, but when dynamic frame read them, they add ".0" at the end of each numeric field(in some cases to date field also). Can some one please help how can I read all the fields in the form of string only so that Dynamic frame dont add trailing zeros
S3bucket_node = glueContext.create_dynamic_frame.from_options(
format_options={"withHeader": True, "separator": "~"},
connection_type="s3",
format="csv",
connection_options={"paths": ["s3://name/{0}".format(file_name_temp)]},
transformation_ctx="S3bucket_node",
)
After creating the S3bucket_node, I am using it to load into Redshift. But as the format of fields is getting changed due to trailing zeros my job is getting failed.
I tried using FromDF and toDF, but could not help to resolve this issue.
Example one of my source file is :
Cust_name, Age, JoiningDate
John,29,20210101
After processing above data from DynamicFrame(S3bucket_node), values get changed as:
John,29.0,20210101.0