Reading CSV using Spark from Zeppelin in EMR

I have a simple CSV file in S3 where I have read it many times using Spark in EMR.

Now I want to use Zeppelin so, I can do some analysis.

My code is very simple

val path="s3://somewhere/some.csv"

val df=
  _spark
    .read
    .format("csv")
    .option("delimiter", "\t")
    .option("header", false)
    .option("mode", ParseModes.DROP_MALFORMED_MODE)
    .option("nullValue", "NULL")
    .option("charset", "UTF-8")
    .option("treatEmptyValuesAsNulls", "true")
    .option("inferSchema", "true")
    .load(path)

But when I try to collect the dataframe

df.collect

I get an error

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateFormat; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2

which is the different versions commons-lang3 between Zeppelin and Spark use.

reference: http://apache-zeppelin-users-incubating-mailing-list.75479.x6.nabble.com/InvalidClassException-using-Zeppelin-master-and-spark-2-1-on-a-standalone-spark-cluster-td4900.html

I have used many different EMR version from 5.3.1 to 5.7.0

I have tried to add in --jars in spark

commons-lang3-3.4.jar 

but with no luck.

Has anyone, had the same error?