Spark unit testing not working with powermockito

Question

I am writing unit tests for my spark application. I used powermockito to test a transformation. When I run the junit test I got error like,

2016-03-24 17:10:10:805 INFO  - Running Spark version 1.6.0
2016-03-24 17:10:11:479 WARN  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-03-24 17:10:11:586 ERROR - Can't find user in Subject:
    Principal: NTUserPrincipal: My_userNAME
    Principal: NTSidUserPrincipal: S-1-5-21-1082354718-1040695487-3441060932-16865
    Principal: NTDomainPrincipal: MY_DOMAIN
    .............
    ...........
    Public Credential: NTNumericCredential: 1872

When I remove the udage of powermockito the above error is gone. I don't know why powermockito is not working on spark rdd transformation.

user_5023032 · Answer

I was also experiencing the same problem. I found the solution, The main problem was with PowerMockRunner and spark session. This is can be rectified by adding

@PowerMockIgnore({"org.apache.hadoop.*","javax.*","com.sun.org.apache.*"})

It worked for me. Still If you face some error , try to figure out what internal classes are being called, and finally ignore it.

user_14933729 · Answer

@PowerMockIgnore() works with the following dependency being added to my build.gradle file

// https://mvnrepository.com/artifact/com.holdenkarau/spark-testing-base testImplementation group: 'com.holdenkarau', name: 'spark-testing-base_2.11', version: '2.1.0_0.7.4'

Spark unit testing not working with powermockito

2 Answers