Struggling to find an efficient way to create SparseMatrix from SparseVectors column. Looking for suggestions/help to convert SparseVector column to SparseMatrix in PySpark:
from pyspark.ml.linalg import SparseVector
from operator import attrgetter
df = sc.parallelize([
(SparseVector(3, [0, 2], [1.0, 3.0]), ),
(SparseVector(3, [1], [4.0]), )
]).toDF(["features"])
df.show()
+-------------------+
| features|
+-------------------+
|(3,[0,2],[1.0,3.0])|
| (3,[1],[4.0])|
+-------------------+
Note: As far I do see from docs Sparse Matrix stored in CSC format. I'm looking for a simple and distributed approach to convert (I can create Scipy CSC using rdd and would like to avoid collecting to driver since is very large matrix).