I built a Raspberry Pi 6 node cluster running Hadoop (HDFS) and Spark. I originally had Spark 2.4.3 and Hadoop 3.2.1 and Scala 2.11 working like a charm. However, I recently upgraded Spark to 3.0.1 and Scala to 2.12, I left Hadoop alone.
When I run dataframe operations it works like a charm; however, when I try to run Spark SQL command they end up erroring out. e.g spark.sql("s e l ec t count(*) from mytable"). The data set I am using is trivially small (a few kb) but yet the Spark SQL errors out. If I do operations using the dataframe api syntax it works like a charm.
I've attached the stderror (Link to text file in Google Drive) for one such spark sql error. Any and all help would be greatly appreciated.