I have a usecase where I need to convert the existing columns of a dataframe into JSON and store in only on one column.
So far I tried this:
import pandas as pd
import json
df=pd.DataFrame([{'a':'sjdfb','b':'jsfubs'},{'a':'ouhbsdv','b':'cm osdn'}]) #Random data
jsonresult1=df.to_json(orient='records')
# '[{"a":"sjdfb","b":"jsfubs"},{"a":"ouhbsdv","b":"cm osdn"}]'
But I want the data to be just string representation of the dictionary and not the list. So I tried this:
>>>jsonresult2=df.to_dict(orient='records')
>>>jsonresult2
# [{'a': 'sjdfb', 'b': 'jsfubs'}, {'a': 'ouhbsdv', 'b': 'cm osdn'}]
This is what I wanted the data to look like but when I try to make this a dataframe the dataframe again is in the format of 2 columns [a,b]. The string representation of these dictionary objects will insert into the dataframe the column data in the required format.
>>>for i in range(len(jsonresult2)):
... jsonresult3.append(str(jsonresult2[i]))
...
>>> jsonresult3
["{'a': 'sjdfb', 'b': 'jsfubs'}", "{'a': 'ouhbsdv', 'b': 'cm osdn'}"]
This is exactly what I wanted. And when I push this to a dataframe I get:
>>> df1
0
++++++++++++++++++++++++++++++++++++
0 | {'a': 'sjdfb', 'b': 'jsfubs'}
1 |{'a': 'ouhbsdv', 'b': 'cm osdn'}
But I feel this is a very inefficient way. How do I make it look and work in an optimized way? My data can exceed 10M rows. And this is taking too long.