Is it possible to programmatically create dynamic frame with groupFiles passed to from_catalog?

user_7231063 0
Viewed 888

I posted this same question on aws forums, but given the poor experience of getting an answer from there I am trying my luck out here.

I have a crawler that is creating a table and a glue job that is creating a dynamic frame from that table. I see that page https://docs.aws.amazon.com/glue/latest/dg/grouping-input-files.html mentions that groupFiles can be specified in table properties or passed as a parameter to create_dynamic_frame.from_options if reading directly from s3.

This architecture blueprint is utilized in many places in our production workload, thus I am searching for alternatives for the above. I wonder if I can pass the same groupFiles:"inPartition" property to additional_options attribute of create_dynamic_frame.from_catalog and if it will work?

The aws documentation is vague on this and I cannot see examples anywhere of people doing this.

Edit: The AWS documentation implies that this is possible, but then the way the documentation pages are linked to each other does not give a full or explicit picture of their features. Thus the question.

aws-glue-data-catalog aws-glue
2 Answers

additional_options is unfortunately nowhere really documented. You would have to ask the AWS Support directly. I contacted them some time ago, if they could update their documentation with all the possible values you could put into additional_options and they said that they would be working on it.

user_126381180

It's not documented anywhere, but you should be able to pass in those parameters like this:

  DataSource0 = glueContext.create_dynamic_frame.from_catalog(
            database = "database-name",
            table_name= "table-name",
            additional_options={"groupFiles": "inPartition", "groupSize": "10485760"},
            transformation_ctx="DataSource0"
  )

It seems to be working for me.

user_110624750
Related