Can pyspark.ml.stat.Summarizer return sparse vector results?

Viewed 81

computing aggregations of sparse vectors using pyspark.ml.stat.Summarizer returns dense vector results - is there a way to force sparse vector operations?

(just converting the result isn't a good solution as the intermediate results will waste a lot of resources)

code to reproduce:

import pyspark
from pyspark.sql.functions import col
from pyspark.ml.stat import Summarizer
from pyspark.ml.linalg import SparseVector, DenseVector

sc = pyspark.SparkContext.getOrCreate()
sql_context = pyspark.SQLContext(sc)

df = sc.parallelize([ ( SparseVector(100, {1: 1.0}),)]).toDF(['v'])
print(df.head())
print(df.select(Summarizer.mean(col('v'))).head())

output:

Row(v=SparseVector(100, {1: 1.0})) 
Row(mean(v)=DenseVector([0.0, 1.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]))

EDIT

an alternative implementation for aggregating sparse vectors would also be relevant

0 Answers
Related