What's the best way to test Spark Java applications?

Viewed 718

Currently, I'm creating classes that use some Datasets from the java Spark API. These datasets are populated from a hive table, using the spark.sql() method.

So, after performing some sql operations (like joins), I have a final dataset.

Right now, I'm struggling on how to write unit test cases for this classes. I can provide the following example of one method of that class:

public Dataset<Row> loadDataSetA() {

    final SparkSession sc; // This has been initialized in another class

    final Dataset<Row> dataSetA = sc.sql("QUERY")
                                .where(upper(col(COL_A)).isin(TYPES));

    final Dataset<Row> dataSetAFinal = dataSetA.select(col(COL_A));

    return dataSetAFinal;
}

I wanted to know if there's a framework or any examples that i can consult to write good test cases for these kind of methods and classes.

Regards

0 Answers
Related