I have a big nested Dataframe with a lots of columns. Here is an extract of the anonymized schema :
df.printSchema()
root
|-- column1: null (nullable = true)
|-- camelCaseColumn1: string (nullable = false)
|-- column2: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- column3: string (nullable = true)
| | |-- camelCaseColumn2: string (nullable = true)
| | |-- column4: string (nullable = true)
| | |-- camelCaseColumn3: struct (nullable = true)
| | | |-- column5: null (nullable = true)
| | | |-- column6: null (nullable = true)
| | |-- camelCaseColumn4: string (nullable = true)
I write the DataFrame to JSON format :
df.write.mode("overwrite").json(targetPath)
And I use copyMerge() function next to merge all the part-files generated :
FileUtil.copyMerge(fs, srcPath, fs, dstFile, deleteSource, configuration, null)
Then when I get the result JSON file for example using an hdfs dfs -cat or -get :
{
"column1":"value",
"camelCaseColumn1":"value",
"column2":[
{
"column3":"value",
"camelcasecolumn2":"value",
"column4":"value",
"camelcasecolumn3":{
"column5":"value",
"column6":"value"
},
"camelcasecolumn4":"value",
We see that the camelCase has been preserved on the first levels of the JSON but on the deeper levels it has been lowercased.
Do you have any explanation and may be a way to preserv camelcase on JSON attributes whatever their level in the file ? We are using Spark 1.6.3 on our environment.
EDIT : Found a solution, see comment below.