I am trying to do a RandomForest Regression in Pyspark. Because most of variables are categorical with many factor levels, I had to string index them, one-hot encode and create a column of feature vectors using Vectorassembler. The output of Vectorassembler is the creation of a 'features' column in the training dataframe. Let's just call this dataframe as train_OHE after doing all the above transformations.
My code works fine till there.
Now when I run the randomforest regressor, there seems to be a strange error. scala.MatchError: [null,(1598,[0,2,234,274,295,1596],[1.0,1.0,1.0,1.0,1.0,1.0])] (of class org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema)
When I run train_OHE.show(), all of the first 20 elements in the 'features' column, look like [1598,[0,2,234,274,295,1596],[1.0,1.0,1.0,1.0,1.0,1.0]] - which is acceptable because of the sparse vector representation.
I think it is because of the extra null element somewhere in one of the rows.
So,
My question is how do I find out which row in my dataframe is causing this problem. Had it been a nomral string, I would have used train_OHE['features'].like('%null%'). In this case of sparse vector elements, how can find out this malfunctioning row? I tried train_OHE['features'].split(',').like('%null%') but errors throw up.