JAVA Spark job Tungsten error when using javaRDD to Dataframe.

Viewed 93

We are trying to create a dataframe using two different methods.

Environment :Spark 1.6.1

Method1 :

JavaRDD<String> javardd = some function.  <This will return json strings>
DataFrame df1= hiveCtx.read().json(javardd)

Method2:

 JavaRDD<String> javardd = some function; <This will return json strings>
 javardd.saveAsTextFile(path);
 DataFrame df1= hiveCtx.read().json(path);

With the DataFrame df1 we are doing joins and other complex function

With Method2 everything works perfectly fine.

However Method1 throws multiple errors. Some times it is

java.io.IOException: Failed to send RPC 
java.nio.channels.ClosedChannelException

some times we get

  org.apache.spark.sql.catalyst.errors.package: execute, tree:
TungstenAggregate(key=[xxxxx

Please let me know if I am doing anything incorrect.

Note: No change in data has been done. It is the same data. No change in lower part of the code, apart from what is mentioned. No change in environment.

0 Answers
Related