Nested list of dictionary with nested list of dictionary into a Pandas dataframe

user_14242377 0
Viewed 209

I need help with converting a nested list of dictionaries with a nested list of dictionaries inside of it to a dataframe. At the end, I want something that looks like (the dots are for other columns in between):

    id       | isbn     | isbn13      | .... | average_rating|
    30278752 |1594634025|9781594634024| .... |3.92           |
    34006942 |1501173219|9781501173219| .... |4.33           |

    review_stat =[{'books': [{'id': 30278752,
                              'isbn': '1594634025',
                              'isbn13': '9781594634024',
                              'ratings_count': 4832,
                              'reviews_count': 8435,
                              'text_reviews_count': 417,
                              'work_ratings_count': 2081902,
                              'work_reviews_count': 3313007,
                              'work_text_reviews_count': 109912,
                             'average_rating': '3.92'}]},
                 {'books': [{'id': 34006942,
                             'isbn': '1501173219',
                             'isbn13': '9781501173219',
                             'ratings_count': 4373,
                             'reviews_count': 10741,
                             'text_reviews_count': 565,
                             'work_ratings_count': 1005504,
                             'work_reviews_count': 2142280,
                             'work_text_reviews_count': 75053,
                             'average_rating': '4.33'}]}]
pandas nested-lists python
4 Answers

If you key is always books

pd.concat([pd.DataFrame(i['books']) for i in review_stat])

         id        isbn         isbn13  ratings_count  reviews_count  text_reviews_count  work_ratings_count  work_reviews_count  work_text_reviews_count average_rating
0  30278752  1594634025  9781594634024           4832           8435                 417             2081902             3313007                   109912           3.92
0  34006942  1501173219  9781501173219           4373          10741                 565             1005504             2142280                    75053           4.33

You can always reset the index if you need

user_60319950

You can also use json_normalize here:

df = pd.json_normalize(review_stat, 'books')

[out]

         id        isbn  ... work_text_reviews_count  average_rating
0  30278752  1594634025  ...                  109912            3.92
1  34006942  1501173219  ...                   75053            4.33
user_102015800

I believe a faster way without needing to append dataframes is to "flatten" the lists, because the dictionary contains single-key books which also contains one element. Therefore, it should easy to flatten into a single list which can be passed to pd.DataFrame:

df = pd.DataFrame([x['books'][0] for x in review_stat])

Outputs:

             id        isbn  ... work_text_reviews_count  average_rating
0  30278752  1594634025  ...                  109912            3.92
1  34006942  1501173219  ...                   75053            4.33
user_118970070

The following should work:

d=[i['books'][0] for i in review_stat]

df=pd.DataFrame(d)
user_82285580
Related