Using either the DataFrame API (df.groupby('id').sum()) or Spark SQL (spark.sql('select * from tableA')) we can build complex queries. I'm writing some code that leverages CASE/when() and I recall that there's a limit to Spark's query size.
I assume there is a size limit to both a Spark Job and a Spark Stage. By "size limit" I mean the length of the instruction that can be submitted / stored / evaluated / distributed etc. The RDBMS equivalent would be a maximum query length, e.g. https://dba.stackexchange.com/questions/131399/is-there-a-maximum-length-constraint-for-a-postgres-query
What's the maximum query size / length in Spark? I have seen it hit before back in 1.3 or 1.5, but (a) can't find it, and (b) it's probably outdated anyway. Is this limit determined by the datatype which stores the DAG?