How to change the schema of a DataFrame (to fix the names of some nested fields)?

I have an issue where when we load a Json file into Spark, store it as Parquet and then try and access the Parquet file from Impala; Impala complains about the names of the columns as they contain characters which are illegal in SQL.

One of the "features" of the JSON files is that they don't have a predefined schema. I want Spark to create the schema, and then I have to modify the field names that have illegal characters.

My first thought was to use withColumnRenamed on the names of the fields in the DataFrame but this only works on top level fields I believe, so I could not use that as the Json contains nested data.

So I created the following code to recreate the DataFrames schema, going recursively through the structure. And then I use that new schema to recreate the DataFrame.

(Code updated with Jacek's suggested improvment of using the Scala copy constructor.)

def replaceIllegal(s: String): String = s.replace("-", "_").replace("&", "_").replace("\"", "_").replace("[", "_").replace("[", "_")
def removeIllegalCharsInColumnNames(schema: StructType): StructType = {
  StructType(schema.fields.map { field =>
    field.dataType match {
      case struct: StructType =>
        field.copy(name = replaceIllegal(field.name), dataType = removeIllegalCharsInColumnNames(struct))
      case _ =>
        field.copy(name = replaceIllegal(field.name))
    }
  })
}

sparkSession.createDataFrame(df.rdd, removeIllegalCharsInColumnNames(df.schema))

This works. But is there a better / simpler way to achive what I want to do?

And is there a better way to replace the existing schema on a DataFrame? The following code did not work:

df.select($"*".cast(removeIllegalCharsInColumnNames(df.schema)))

It gives this error:

org.apache.spark.sql.AnalysisException: Invalid usage of '*' in expression 'cast'