I am facing an issue while joining two data frames based on a column. I am getting a Stackoverflow error but am unable to find out the reason behind that. Is there any way to know why facing the same...
var joinedDf = event.join(event_status_df, Seq("_id"), "left")
I did a count of both data frames first one has 150 counts and the second, has 15, the first data frame has 80 columns whereas the second data frame has only 5 columns.
This join gives me
Exception in thread "main" java.lang.StackOverflowError
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.findAliases(Analyzer.scala:1517)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.collectConflictPlans$1(Analyzer.scala:1183)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.$anonfun$dedupRight$7(Analyzer.scala:1217)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.collectConflictPlans$1(Analyzer.scala:1217)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences$.$anonfun$dedupRight$7(Analyzer.scala:1217)
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.TraversableLike.flatMap(TraversableLike.scala:245)
at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:242)
at scala.collection.immutable.List.flatMap(List.scala:355)
I have the first data frame i.e event DF cached but not sure why the above error. I get that by providing -Xss value I can resolve the issue but want to understand why the issue is. Any ideas or thoughts which I can check and resolve in another way?