create a spark dataframe from a nested json file in scala

Viewed 20183

I have a json file that looks like this

{
"group" : {},
"lang" : [ 
    [ 1, "scala", "functional" ], 
    [ 2, "java","object" ], 
    [ 3, "py","interpreted" ]
]
}

I tried to create a dataframe using

val path = "some/path/to/jsonFile.json"
val df = sqlContext.read.json(path)
df.show()

when I run this I get

df: org.apache.spark.sql.DataFrame = [_corrupt_record: string]

How do we create a df based on contents of "lang" key? I do not care about group{} all I need is, pull data out of "lang" and apply case class like this

case class ProgLang (id: Int, lang: String, type: String )

I have read this post Reading JSON with Apache Spark - `corrupt_record` and understand that each record needs to be on a newline but in my case I cannot change the file structure

2 Answers
Related