Iterate over AWS Glue DynamicFrame

user_3005674 0
Viewed 3738

I'm quite new to AWS Glue and still trying to figure things out, I've tried googling the following but can't find an answer...

Does anyone know how to iterate over a DynamicFrame in an AWS Glue job script?

For example, I'm trying to do the following:

datasource0 = glueContext.create_dynamic_frame.from_catalog(
                        database = "...", 
                        table_name = "...", 
                        transformation_ctx = "datasource0")

for r in datasource0:
    print(r)

But receive the following error:

'DynamicFrame' object is not iterable
Traceback (most recent call last):
TypeError: 'DynamicFrame' object is not iterable
aws-glue amazon-web-services
1 Answers

Convert Glue's DynamicFrame into Spark's DataFrame and use foreach function to iterate rows:

def f(row):
   print(row.name)

...

datasource0.toDF().foreach(f)
user_17915100
Related