Can I use multi-threading inside pyspark mapPartitions?

user_1686903 0
Viewed 329

Can I use multi-threading inside pyspark mapPartitions?

I am running a spark job, where I have to do API calls over each row. We are using rdd map to run a python function to do API calls. These API calls take 0.01 ~ 0.5 seconds.

Can we use rdd mapParitions to run a python multi-threading function? What are the implications or Why can't we use threading?

SparkForApiCalls

Note: We are aware that, Spark may not the best tool for this job. Also, we can't use pandas udf or vectorization due to PyArrow dependencies.

Image Source: https://medium.com/ibm-data-science-experience/using-spark-as-a-parallel-processing-framework-for-accessing-rest-based-data-services-cd4c98526784 (not related to this question)

pyspark python
1 Answers

You should be able to, provided the target system is designed to cope with so many calls.

I would recommend going for non-blocking IO based API calls, those should be much more CPU efficient. If you must assign more CPUs for task in hand, then you can instruct spark to do so with spark.task.cpus. Beware, this is a setting at spark application level. This could actually make your application slow if you have long data processing pipeline.

user_13407670
Related