I am trying to use to_json() from the spark with below data set
| Column a | Column b | Column c |
|---|---|---|
| record1 | value1 | {'op':'and','value':[{'id':1,'value':'name'}]} |
| record2 | value2 | {'op':'and','value':[{'op':'and':value:[{'id':1,'value':'name'}]}]} |
I am looking for the conversion to the resultant set with
| Column d |
|---|
| {'column a':record1,'column b':value1,'Column c':{'op':'and','value':[{'id':1,'value':'name'}]} } |
| {'column a':record2,'column b':value2,'Column c':{'op':'and','value':[{'op':'and':value:[{'id':1,'value':'name'}]}]} |
To achieve this resultant i am using to_json for the conversion, but i am getting the result in the below mentioned format. to_json that I used is
to_json(struct(col("column a"), col("column b"), col("column c")))
{"column a":record2,"column b":value2,"Column c":"{\"op\":\"and\",\"value\":[{\"op\":\"and\":\"value\":[{\"id\":1,\"value\":\"name\"}]}]}
So i tried whether should i use from_json to but i am having the circular reference, I tried to use defining the schema with StructType i am not able to define it, i used Java Bean to define the reference but getting error.
Could anyone let us know how to use the to_json which results the column C to have object instead of String Type