pyspark image dimension reduction with PCA

Viewed 20

I am using Pyspark in AWS cloud to extract the image features:

ImageSchema.imageFields
img2vec = F.udf(lambda x: DenseVector(ImageSchema.toNDArray(x).flatten()),
                VectorUDT())
df_vec = df_cat.withColumn('original_vectors', img2vec("image"))
df_vec.show()

After having standardized the data:

standardizer = MinMaxScaler(inputCol="original_vectors",
                            outputCol="scaledFeatures",
                            min=-1.0,
                            max=1.0)
#withStd=True, withMean=True)
model_std = standardizer.fit(df_vec)
df_std = model_std.transform(df_vec)
df_std.show()

... when I apply PCA for dimension reduction, I receive an error that I could not debug for a couple of weeks :(

Error_1

Error_2

Could you please help me to solve that?

I use Pyspark spark-3.0.3-bin-hadoop2.7

0 Answers
Related