Spark SQL - Get Column Names of a Hive Table in a String

user_3725491 0
Viewed 6194

I'm trying to get the column names of a Hive table in a comma separated String. This is what I'm doing

val colNameDF = spark.sql("show columns in hive_table")
val colNameStr = colNameDF.select("col_name").collect.mkString(", ")

And the output I get is

res0: String = [col_1], [col_2], [col_3]

But what I want is col_1, col_2, col_3. I can remove [ and ] from the String, but I'm curious as to whether we can get the column names without the brackets in the first place.

Edit: The column names in the Hive table don't contain [ ]

apache-spark-sql apache-spark hive
3 Answers

Instead of show columns, Try below approach as it is faster than yours.

val colNameDF = spark.sql("select * from hive_table").limit(0)

Or

val colNameDF = spark.table("hive_table").limit(0)

val colNameStr = colNameDF.columns.mkString(", ")
user_85934140

The collect returns to you an array of Row which is particularly represented internally as array of values, so you need to trick it like this:

val colNameDF = spark.sql("show columns in hive_table")
val colNameStr = colNameDF.select("col_name").collect.map(r=>r.getString(0)).mkString(", ")
user_11715900

Building on @Srinivas' answer above, here is the equivalent Python code. It is very fast:

colNameStr = ",".join(spark.table(hive_table).limit(0).columns)
user_93472310
Related