How to run pySpark code on multiple workers in AWS glue

user_1248779 0
Viewed 187

I have a Glue Table. The table has been created by crawling an S3 bucket that contains parquet files. The data is partitioned by year/month/day.

I fetch the Dynamic Frame of a specific partition first. That’s easy.

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "testdata", table_name = "sampletable", transformation_ctx = "datasource0",push_down_predicate = "(year == '2019' and month == '08' and day == '02')")

I want to find the max value of a specific column. That’s straightforward as well:

max = datasource0.toDF().agg({“my-col”:”max”}).first()[0]

When I launched this job, I selected 10 workers (the default).

  • Is the create_dynamic_frame being processed on 10 worker?

  • Is the agg function being processed on 10 workers?

If the answer is yes, how can I get this intuition? If the answer is no, then, how can I make it so?

My objective is to speed up the execution of the above code, which runs today in 5 min.

aws-glue pyspark apache-spark
1 Answers

I think when working with dataframes, the glue spark cluster will handle the parallelization.

You can get a count of partitions using:

datasource0.rdd.getNumPartitions()

If you are working with custom functions (UDFs), you can in the distributed functions get a handle to the execution context:

ctx = TaskContext()
logger.warning(f"PartitionID: {ctx.partitionId()}")
user_3768030
Related