What is a good way to aggregate dataframes in AWS Glue

user_8614203 0
Viewed 1710

I am trying to aggregate dataframes in AWS Glue. I have used the following pySpark code to perform the aggregation:

mydataframe.groupby('id').agg({'value', 'operation'})

Is there a better way to perform the aggregation in AWS Glue?

Thanks!

spark-dataframe aws-glue apache-spark amazon-web-services
1 Answers

I convert it to Dataframe using pyspark and do the aggregation, then converting it back to Dynamicframe. However, it seems that you need to manually add partitions to the dataframes.

user_67888690
Related