Is it possible to make a generic training pipeline for Random Forest in Spark ML?

Viewed 307

I am just starting out with Spark and Spark ML and I am finding it much more difficult than Python and sklearn.

The development time is much greater so I am wondering if it's feasible to make a generic pipeline that works on any (sufficiently small) dataset and trains a Random Forest Classifier. Ideally, I would create a function like

def trainClassifier(df: DataFrame, labelColumn: String) {
  ...
}

A lot of the development time in Spark is spent on encoding columns to numerical columns and then forming a vector out of the features so that Spark ML's Random Forest can actually work with it. So one ends up writing lines like

val indexer = new StringIndexer()
                   .setInputCol("category")
                   .setOutputCol("categoryIndex")
                   .fit(df)

val indexed = indexer.transform(df)

val encoder = new OneHotEncoder()
                   .setInputCol("categoryIndex")
                   .setOutputCol("categoryVec")

val encoded = encoder.transform(indexed)

So my question is more of a design question (please direct me to a different site, if appropriate) about how to write a generic training function for classification that would work with any DataFrame, but it is also a question about Spark in the sense that I am asking whether or not this sort of thing is feasible within Spark (so this is kind of an API question and so it's a better fit for stackoverflow)?

EDIT: What I mean is that I do not what to specify the columns and manually convert the columns for every new dataframe. I want a function trainClassifier which will take in a variety of dataframes with different columns and different column types. Something that iterates over all columns except the labelColumn and compiles those together into a feature vector that the classifier can use.

1 Answers
Related