Scala Dataframe columns with space save as a databricks table

user_9235711 0
Viewed 365

I am working on databricks notebook (Scala) and I have a spark query that goes kinda like this:

df = spark.sql("SELECT columnName AS `Column Name` FROM table")

I want to store this as a databricks table. I tried below code for the same:

df.write.mode("overwrite").saveAsTable("df")

But it is giving an error because of the space in the column name. Here's the error:

Attribute name contains invalid character(s) among " ,;{}()\n\t=". Please use alias to rename it.;

I don't want to remove the space so is there any alternative for this?

databricks apache-spark-sql apache-spark scala
1 Answers

No, that's a limitation of the underlying technologies used by Databricks under the hood (for example, PARQUET-677). The only solution here is to rename column, and if you need to have space in the name, do renaming when reading it back.

Alex_Ott_186270
Related