Unit testing with Spark dataframes

user_3510671 0
Viewed 21254

I'm trying to test a part of my program which performs transformations on dataframes I want to test several different variations of these dataframe which rules out the option of reading a specific DF from a file

And so my questions are:

  1. Is there any good tutorial on how to perform unit testing with Spark and dataframes, especially regarding the dataframes creation?
  2. How can I create these different several lines dataframes without a lot of boilerplate and without reading these from a file?
  3. Are there any utility classes for checking for specific values inside a dataframe?

I obviously googled that before but could not find anything which was very useful. Among the more useful links I found were:

It would be great if examples/tutorials are in Scala but I'll take whatever language you've got

Thanks in advance

apache-spark-sql apache-spark scala unit-testing
3 Answers

For those looking to achieve something similar in Java, you can use start by using this project to initialize a SparkContext within your unit tests: https://github.com/holdenk/spark-testing-base

I personally had to mimick the file structure of some AVRO files. So I used Avro-tools (https://avro.apache.org/docs/1.8.2/gettingstartedjava.html#download_install) to extract the schema from my binary records using the following command:

java -jar $AVRO_HOME/avro tojson largeAvroFile.avro | head -3

Then, using this small helper method, you can convert the output JSON into a DataFrame to use in your unit tests.

private DataFrame getDataFrameFromList() {
    SQLContext sqlContext = new SQLContext(jsc());
    ImmutableList<String> elements = ImmutableList.of(
        {"header":{"appId":"myAppId1","clientIp":"10.22.63.3","createdDate":"2017-05-10T02:09:59.984Z"}}
        {"header":{"appId":"myAppId1","clientIp":"11.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
        {"header":{"appId":"myAppId1","clientIp":"12.22.63.3","createdDate":"2017-05-11T02:09:59.984Z"}}
    );
    JavaRDD<String> parallelize = jsc().parallelize(elements);
    return sqlContext.read().json(parallelize);
}
user_37992900
user_84423660
Related