web scraping - nested dict

user_17277677 0
Viewed 51

I am looking for a way to transform nested dictionary to dataframe

import requests
import pandas as pd
from bs4 import BeautifulSoup
r = requests.get('https://olxdata.azurewebsites.net/olx?city=wszystkie&dataType=mieszkania')
soup = BeautifulSoup(r.content, 'html.parser')
print(soup)

As you could see the format is as below, a dictionary with list of dictionaries as a value:

{"olx":[{"date":"2020-04-30T00:00:00","toSell":27964,"toRent":41664}(...)], "otodom":[{"date":"2020-04-30T00:00:00","toSell":27964,"toRent":41664}(...)]

There are two elements in this dictionary - ideally I would like them to be an index like below:

date toSell toRent
olx
otodom
web-scraping dictionary nested python
1 Answers

You could do something like this:

import requests
import pandas as pd

r = requests.get('https://olxdata.azurewebsites.net/olx?city=wszystkie&dataType=mieszkania')
df = pd.DataFrame()
for x in r.json():
    s_df = pd.DataFrame(r.json()[x])
    s_df['type_of'] = x
    df = pd.concat([df, s_df], axis=0)
print(df)

This would return:

date    toSell  toRent  type_of
0   2020-04-30T00:00:00 27964   41664   olx
1   2020-05-01T00:00:00 28071   41678   olx
2   2020-05-02T00:00:00 27884   41646   olx
3   2020-05-03T00:00:00 27724   41574   olx
4   2020-05-04T00:00:00 28021   42299   olx
... ... ... ... ...
842 2022-08-20T00:00:00 154562  12675   otodom
843 2022-08-21T00:00:00 154157  12753   otodom
844 2022-08-22T00:00:00 153991  12488   otodom
845 2022-08-23T00:00:00 154419  12699   otodom
846 2022-08-24T00:00:00 154728  12772   otodom
user_194751850
Related