PySpark filter by value at given SparseVector() index

user_4671049 0
Viewed 624

I'm having issues trying to do something that seems like it would be very simple. I have a DF in the following format:

+---------+---------------------+
|id       |feat_vctr            |
+---------+---------------------+
|XXXXXXXX |(4,[],[])            |
|XXXXXXXX |(4,[0],[5.0])        |
|XXXXXXXX |(4,[2,3],[25.0,15.0])|
+---------+---------------------+

where feat_vctr is pyspark.ml.linalg.SparseVector.

Note that printSchema() shows this simply as a Vector, however it is in the format of a sparse vector...

Anyways I'd like to filter this into 4 DF's, where each dataframe is a filtered version of the above, where all rows that have no value at a given index are filtered out.

I'm trying to use:

filtered_df_idx_0 = df.filter(df.feat_vctr[0] > 0.0)
filtered_df_idx_1 = df.filter(df.feat_vctr[1] > 0.0)
filtered_df_idx_2 = df.filter(df.feat_vctr[2] > 0.0)
filtered_df_idx_3 = df.filter(df.feat_vctr[3] > 0.0)

I'm getting an error

Py4JJavaError: An error occurred while calling o1089.filter.
: org.apache.spark.sql.AnalysisException: Can't extract value from feat_vctr#1007: need struct type but got struct<type:tinyint,size:int,indices:array<int>,values:array<double>>;

It's strange to me because I can create a sparse vector directly using:

v = SparseVector(4, {1:2.3, 3:4.0})

v[1]

Which returns 2.3 (in a jupyter notebook)

pyspark apache-spark-sql apache-spark python
1 Answers

I was not able to do that inside the filter function

it seems it has to use UDF to achieve this:

# Filter the empty Sparse Vector
def no_empty_vector(value):
    if value.indices.size > 0:
        return True
    else:
        return False


no_empty_vector_udf = udf(no_empty_vector, BooleanType())
df = df.filter(no_empty_vector_udf('features'))
df.show()
user_61696880
Related