I saw that a general recommendation for anyone using spark (in my case with Scala) is to avoid any action that gets all data from executers to driver (collect, count, sum etc). However, when I tried to use spark statistic library http://spark.apache.org/docs/2.2.0/ml-statistics.html I found out that the correlation matrix and ChiSquareTest methods expect array or matrix obtained of array\seq as their parameters, hence I do not see how I could avoid do collect to the dataframe (and some more manipulation I assume to make it of Vectors not Row type) if I want to use this functions. Will appreciate any help.
1 Answers
- In your example, both
Correlation.corrandChiSquareTest.testare accepting a dataframe, so you don't need to collect data before passing it to these functions. Results of these functions you'll have to collect on the driver, but that shouldn't cause any issues, as output size should be much smaller than the initial dataset, and it should easily fit into the driver's memory. - To your question in the comment about
groupBy/joins- those are "expensive", but for a different reason. Grouping and joins leads to data shuffling - so, your workers would need to send a lot of data across the network, which takes much more time, than local data processing. Though, still, if you have to do this - it's ok to do it, just be aware of performance implications. collectmethod is not recommended to use on a full dataset, as it may lead to an OOM error on the driver (imagine, that you had 50 Gb dataset, distributed over a cluster, and now you're collecting it on a single node), but if you've already processed your data, and you know, that there would be some reasonable amount of rows - it's pretty safe to do it.countshould not be a problem from a memory standpoint at all, as it just returns number of rows in your dataset, instead of sending all of them to a driver node.
Related