I have a huge dataset and am using Apache Spark for data processing.
Using Apache Arrow, we can convert Spark-compatible data-frame to Pandas-compatible data-frame and run operations on it.
By converting the data-frame, will it achieve the performance of parallel processing seen in Spark or will it behave like Pandas?
As you can see on the documentation here
Note that even with Arrow, toPandas() results in the collection of all records in the DataFrame to the driver program and should be done on a small subset of the data
The data will be sent to the driver when the data is moved to the Pandas data frame. That means that you may have performance issues if there is too much data for the driver to deal with. For that reason, if you are decided to use Pandas, try to group the data before calling to toPandas() method.
It won't have the same parallelization once it's converted to a Pandas data frame because Spark executors won't be working on that scenario. The beauty of Arrow is to be able to move from the Spark data frame to Pandas directly, but you have to think on the size of the data
Another possibility would be to use other frameworks like Koalas. It has some of the "beauties" of Pandas but it's integrated into Spark.
