computing aggregations of sparse vectors using pyspark.ml.stat.Summarizer returns dense vector results - is there a way to force sparse vector operations?
(just converting the result isn't a good solution as the intermediate results will waste a lot of resources)
code to reproduce:
import pyspark
from pyspark.sql.functions import col
from pyspark.ml.stat import Summarizer
from pyspark.ml.linalg import SparseVector, DenseVector
sc = pyspark.SparkContext.getOrCreate()
sql_context = pyspark.SQLContext(sc)
df = sc.parallelize([ ( SparseVector(100, {1: 1.0}),)]).toDF(['v'])
print(df.head())
print(df.select(Summarizer.mean(col('v'))).head())
output:
Row(v=SparseVector(100, {1: 1.0}))
Row(mean(v)=DenseVector([0.0, 1.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]))
EDIT
an alternative implementation for aggregating sparse vectors would also be relevant