How to ignore the columns if it is not present in a dataframe using spark-SQL?

user_18463160 0
Viewed 387

Below is the dataframe from which I need to derive the output.

|A|B|C|E|F|
+---------+
|1|2|3|5|6| 
+---------+

Dataset<Row> set = spark.sql("select A as one, B as two, C as three ,D as four, E as five, F as six from input");

I need to skip the column 'D' if it is not present and move on to the other column(i.e) E.

If any of the column is not present then should print null and move on to the other column assignment.

apache-spark-sql apache-spark dataframe
1 Answers

you can try and check first which columns exist in the input, for example:

input = spark.read.parquet(...)
cols = input.columns // returns an Array[String]

and then format your query accordingly.

user_116178560
Related