My project includes many operations between numpy arrays and numpy matrices that are currently performed within UDF, do you think if we used the internal structures in PySpark, we would have an increase in performance? (matrix --> dataframe, numpy array --> dense vectors) Thank you!
1 Answers
UDFs are generally slower than pyspark.sql.functions working on DataFrame API, you should generally avoid those as much as possible due to serialisation deserialisation overhead.
Related