scala dataframe column names replace '-' with _ for nested json

Viewed 33

I am working with Nested json, using scala and need to replace the - in column names with _.

Schema of json:

 |-- a-type: struct (nullable = true)
 |    |-- x-Type: array (nullable = true)
 |    |    |-- element: string (containsNull = true)
 |    |-- part: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- x-Type: array (nullable = true)
 |    |    |    |    |-- element: string (containsNull = true)
 |    |    |    |-- Length: long (nullable = true)
 |    |    |    |-- Order: long (nullable = true)
 |    |    |    |-- y-Name: string (nullable = true)
 |    |    |    |-- Payload-Text: string (nullable = true)
 |    |-- Date: string (nullable = true)

I am using below code which only works at first level. However, I have to replace - with _ at all levels. Any help is really appreciated.

Code used currently:

scJsonDF.columns.foreach { col =>
    println(col + " after column replace " + col.replaceAll("-", ""))
    scJsonDFCorrectedCols = scJsonDFCorrectedCols.withColumnRenamed(col, col.replaceAll("-", "")
    )
}

I am looking for a dynamic solution as there are different structures available.

1 Answers
Related