Why I can't see ingestion time columns in the output schema in Custom Transform in AWS Glue?

user_19075367 0
Viewed 73

I try to add ingestion time columns to my Dynamic Frame using add_ingestion_time_columns (https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-glue-context.html#aws-glue-api-crawler-pyspark-extensions-glue-context-add-ingestion-time-columns). I created a job in AWS Glue where I joined two csv files and then I created a Custom Transform where I try to add these new time columns to my output:

def MyTransform (glueContext, dfc) -> DynamicFrameCollection:
    FirstDataFrame = dfc.select(list(dfc.keys())[0]).toDF()
    dynamic_frame = DynamicFrame.fromDF(glueContext.add_ingestion_time_columns(FirstDataFrame, "hour"), glueContext, "DynamicFrameDateAndHour" )
    return DynamicFrameCollection ({"CustomTrasform": dynamic_frame}, glueContext)

I expected that I can see new columns in the Output schema but it looks like nothing happened. Does anyone know why and what should I change to add this columns? visual job

aws-glue pyspark amazon-web-services
1 Answers

Ok I got this - these new columns we have to add manually and they fill automatically: go to Output schema in your Custom Transform -> Edit -> Add root key -> add first column like ingest_year -> Apply -> again Add root key and add another columns etc.

user_190753670
Related