We are trying to create a dataframe using two different methods.
Environment :Spark 1.6.1
Method1 :
JavaRDD<String> javardd = some function. <This will return json strings>
DataFrame df1= hiveCtx.read().json(javardd)
Method2:
JavaRDD<String> javardd = some function; <This will return json strings>
javardd.saveAsTextFile(path);
DataFrame df1= hiveCtx.read().json(path);
With the DataFrame df1 we are doing joins and other complex function
With Method2 everything works perfectly fine.
However Method1 throws multiple errors. Some times it is
java.io.IOException: Failed to send RPC
java.nio.channels.ClosedChannelException
some times we get
org.apache.spark.sql.catalyst.errors.package: execute, tree:
TungstenAggregate(key=[xxxxx
Please let me know if I am doing anything incorrect.
Note: No change in data has been done. It is the same data. No change in lower part of the code, apart from what is mentioned. No change in environment.