pyspark partition and batch data for api call

I'm trying to do the following task and I'm wondering if this the most efficient way to do so in PySpark

sample data: | email | balance | |---------------------|------------------| | js@google.com | 34 | | ab@yahoo.com | 42 |

The problem I'm trying to solve is to send data to a third party via post request. My solution is:

1. df.repartition(5) // 5 is the max number of concurrent calls
2. foreachPartition(batch_func) divide each partition into batches of 200 records // max 200 records per call

   • def batch_func(iterator):
       batch = [x for x in itertools.islice(iterator, 200)]
       while len(batch) > 0:
           yield batch
           batch = [x for x in itertools.islice(iterator, 200)]
     

3. each call needs to be in the following format:

payload = { "action": "createOrUpdate", 
            "dedupeBy": some_input_field, 
            "input": [{"email":"js@google.com", "money": 34}, {"email": "ab@yahoo.com ", "money": 42}]
}

basically I want to make numOfPartitions API calls in parallel and make the API calls within each partition sequentially (the input to each API call is a batch)

Does anyone have any advice on how to achieve this?

Thanks!