I am using logistic regression to train a dataset with 20,000 features and am also using PCA for dimensionality reduction. The snippet below is directly from the documentation here: https://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html#principal-component-analysis-pca however, I am (understandably) getting Java Heap Space errors (both locally and when running on a GCP cluster). How can I use PCA for such a large feature set, or, what would be a better approach for dimensionality reduction?
import org.apache.spark.mllib.feature.PCA
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.rdd.RDD
val data: RDD[LabeledPoint] = sc.parallelize(Seq(
new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 1)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 1, 0)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0)),
new LabeledPoint(0, Vectors.dense(1, 0, 0, 0, 0)),
new LabeledPoint(1, Vectors.dense(1, 1, 0, 0, 0))))
// Compute the top 5 principal components.
val pca = new PCA(5).fit(data.map(_.features))
// Project vectors to the linear space spanned by the top 5 principal
// components, keeping the label
val projected = data.map(p => p.copy(features = pca.transform(p.features)))