I have a dataframe(s) on which the Spark SQL queries will run. The problem is that the SQL query may have columns which may not be available in the dataframe, or even if the column is present some of the rows in the column may be null.
So the question is, in Spark 2.2 version and scala 2.12 how do I achieve this? An example of nested column is: a.b.c (where b can be a Map/ Array) and default values can be String/Int.