glue etl jobs - get s3 subfolders using create_dynamic_frame.from_options

Question

I am creating an AWS Glue ETL job, but I'm running into some roadblocks with file retrieval.

It seems that the following code only gets the files at the root folder 2017 and not any further. Is there any way to include all subfolders and files within them?

dyf = glueContext.create_dynamic_frame.from_options(
    's3',
    {"paths": [
        's3://bucket/2017/'
        ]},
    "json",
    transformation_ctx = "dyf")

user_3615669 · Answer

Found a solution for this problem, looks like the dictionary accepts more parameters, the one I needed was "recurse". You can also exclude certain patterns with "exclusions".

Source https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect.html#aws-glue-programming-etl-connect-s3

dyf = glueContext.create_dynamic_frame.from_options(
    's3',
    {
        "paths": [
            's3://bucket/2017/'
        ],
        "recurse" : True
    },
    "json",
    transformation_ctx = "dyf")

glue etl jobs - get s3 subfolders using create_dynamic_frame.from_options

1 Answers