i have a question about DynamicFrames, i want to run query in the moment when i get data from the catalog and avoid get all the data and then query. At the moment it's not a lot of data, but maybe in the future. Now, i testing with the parameter "push_down_predicate", but only run when the data is partitioned.
Example:
df_data = glueContext.create_dynamic_frame.from_catalog(database = "datalake-raw",
table_name = "data",
transformation_ctx = "df_data",
push_down_predicate = "( CampaignKey == '1')")
Know another way to run that? is this possible to do?
Thanks!!