json.loads only reads the data of the last page in Python

Question

First I get rawdata using

#api multipage request
import http.client, urllib.request, urllib.parse, urllib.error, base64, pandas as pd, json

page_num=0

headers = {
    # Request headers
    'X-Customer-Id':'ID',
    'Ocp-Apim-Subscription-Key':'key'
}



try:
        conn = http.client.HTTPSConnection('url')
        for page_num in range(2):
            params = urllib.parse.urlencode({'page': page_num,'size':100,})
            conn.request("GET", "/stock/v1/?%s" % params,"{body}", headers)
            response = conn.getresponse()
            rawdata = response.read()
            print(rawdata)
            conn.close()
           
except Exception as e:
        print("[Errno {0}] {1}".format(e.errno, e.strerror))

When I print the rawdata_object I get

b'{"content":[{"sku":"0100094WITB075","ean":"5415015112535","quantity":0},{"sku":"0100375BISD070","ean":"5410760268643","quantity":0},{"sku":"0100375BISD075","ean":"5410760268650","quantity":0},{"sku":"0100375BISD080","ean":"5410760268667","quantity":0},{"sku":"0100375BISD085","ean":"5410760268674","quantity":0},{"sku":"0100375BISD090","ean":"5410760268681","quantity":0},{"sku":"0100375BISE070","ean":"5410760268698","quantity":0},{"sku":"0100375BISE075","ean":"5410760268704","quantity":0},{"sku":"0100375BISE080","ean":"5410760268711","quantity":0},{"sku":"0100375BISE085","ean":"5410760268728","quantity":0},{"sku":"0100375BISE090","ean":"5410760268735","quantity":0},{"sku":"0100377BISB070","ean":"5410760268742","quantity":0},{"sku":"0100377BISB075","ean":"5410760268759","quantity":0},{"sku":"0100377BISB080","ean":"5410760268766","quantity":0},{"sku":"0100377BISB085","ean":"5410760268773","quantity":0},{"sku":"0100377BISC070","ean":"5410760268780","quantity":0},{"sku":"0100377BISC075","ean":"5410760268797","quantity":0},{"sku":"0100377BISC080","ean":"5410760268803","quantity":0},{"sku":"0100377BISC085","ean":"5410760268810","quantity":0},{"sku":"0100377BISD070","ean":"5410760268827","quantity":0},{"sku":"0100377BISD075","ean":"5410760268834","quantity":0},{"sku":"0100377BISD080","ean":"5410760268841","quantity":0},{"sku":"0100377BISD085","ean":"5410760268858","quantity":0},{"sku":"0100377WITB070","ean":"5410760209042","quantity":0},{"sku":"0100377WITB075","ean":"5410760209059","quantity":0},{"sku":"0100377WITB080","ean":"5410760209066","quantity":0},{"sku":"0100377WITB085","ean":"5410760209073","quantity":0},{"sku":"0100377WITC070","ean":"5410760209080","quantity":0},{"sku":"0100377WITC075","ean":"5410760209097","quantity":0},{"sku":"0100377WITC080","ean":"5410760209103","quantity":0},{"sku":"0100377WITC085","ean":"5410760209110","quantity":0},{"sku":"0100377WITD070","ean":"5410760209127","quantity":0},{"sku":"0100377WITD075","ean":"5410760209134","quantity":0},{"sku":"0100377WITD080","ean":"5410760209141","quantity":0},{"sku":"0100377WITD085","ean":"5410760209158","quantity":0},{"sku":"0100410ANAB075","ean":"5415338850121","quantity":5},{"sku":"0100410ANAB080","ean":"5415338850169","quantity":2},{"sku":"0100410ANAB085","ean":"5415338850206","quantity":1},{"sku":"0100410ANAB090","ean":"5415338850244","quantity":0},{"sku":"0100410ANAC075","ean":"5415338850138","quantity":3},{"sku":"0100410ANAC080","ean":"5415338850176","quantity":1},{"sku":"0100410ANAC085","ean":"5415338850213","quantity":0},{"sku":"0100410ANAC090","ean":"5415338850251","quantity":0},{"sku":"0100410ANAD070","ean":"5415338850107","quantity":0},{"sku":"0100410ANAD075","ean":"5415338850145","quantity":0},{"sku":"0100410ANAD080","ean":"5415338850183","quantity":0},{"sku":"0100410ANAD085","ean":"5415338850220","quantity":0},{"sku":"0100410ANAD090","ean":"5415338850268","quantity":0},{"sku":"0100410ANAE070","ean":"5415338850114","quantity":0},{"sku":"0100410ANAE075","ean":"5415338850152","quantity":0},{"sku":"0100410ANAE080","ean":"5415338850190","quantity":0},{"sku":"0100410ANAE085","ean":"5415338850237","quantity":0},{"sku":"0100410ANAE090","ean":"5415338850275","quantity":0},{"sku":"0100410APSB075","ean":"5400977140757","quantity":0,"availabilityDate":"2023-02-24"},{"sku":"0100410APSB080","ean":"5400977140795","quantity":5},{"sku":"0100410APSB085","ean":"5400977140832","quantity":5},{"sku":"0100410APSB090","ean":"5400977140870","quantity":5},{"sku":"0100410APSC075","ean":"5400977140764","quantity":5},{"sku":"0100410APSC080","ean":"5400977140801","quantity":5},{"sku":"0100410APSC085","ean":"5400977140849","quantity":5},{"sku":"0100410APSC090","ean":"5400977140887","quantity":5},{"sku":"0100410APSD070","ean":"5400977140733","quantity":5},{"sku":"0100410APSD075","ean":"5400977140771","quantity":5},{"sku":"0100410APSD080","ean":"5400977140818","quantity":5},{"sku":"0100410APSD085","ean":"5400977140856","quantity":5},{"sku":"0100410APSD090","ean":"5400977140894","quantity":5},{"sku":"0100410APSE070","ean":"5400977140740","quantity":5},{"sku":"0100410APSE075","ean":"5400977140788","quantity":5},{"sku":"0100410APSE080","ean":"5400977140825","quantity":5},{"sku":"0100410APSE085","ean":"5400977140863","quantity":5},{"sku":"0100410APSE090","ean":"5400977140900","quantity":5},{"sku":"0100410ATBB075","ean":"5400977107965","quantity":5},{"sku":"0100410ATBB080","ean":"5400977108009","quantity":5},{"sku":"0100410ATBB085","ean":"5400977108047","quantity":5},{"sku":"0100410ATBB090","ean":"5400977108085","quantity":4},{"sku":"0100410ATBC075","ean":"5400977107972","quantity":5},{"sku":"0100410ATBC080","ean":"5400977108016","quantity":5},{"sku":"0100410ATBC085","ean":"5400977108054","quantity":5},{"sku":"0100410ATBC090","ean":"5400977108092","quantity":5},{"sku":"0100410ATBD070","ean":"5400977107941","quantity":5},{"sku":"0100410ATBD075","ean":"5400977107989","quantity":5},{"sku":"0100410ATBD080","ean":"5400977108023","quantity":5},{"sku":"0100410ATBD085","ean":"5400977108061","quantity":5},{"sku":"0100410ATBD090","ean":"5400977108108","quantity":0},{"sku":"0100410ATBE070","ean":"5400977107958","quantity":5},{"sku":"0100410ATBE075","ean":"5400977107996","quantity":5},{"sku":"0100410ATBE080","ean":"5400977108030","quantity":5},{"sku":"0100410ATBE085","ean":"5400977108078","quantity":5},{"sku":"0100410ATBE090","ean":"5400977108115","quantity":5},{"sku":"0100410BBIB070","ean":"5415015438239","quantity":0},{"sku":"0100410BBIB075","ean":"5415015438277","quantity":0},{"sku":"0100410BBIB080","ean":"5415015438314","quantity":0},{"sku":"0100410BBIB085","ean":"5415015438352","quantity":0},{"sku":"0100410BBIB090","ean":"5415015438390","quantity":0},{"sku":"0100410BBIC070","ean":"5415015438246","quantity":0},{"sku":"0100410BBIC075","ean":"5415015438284","quantity":0},{"sku":"0100410BBIC080","ean":"5415015438321","quantity":0},{"sku":"0100410BBIC085","ean":"5415015438369","quantity":0},{"sku":"0100410BBIC090","ean":"5415015438406","quantity":0},{"sku":"0100410BBID070","ean":"5415015438253","quantity":0}],"totalElements":282582,"size":100,"number":0}'
b'{"content":[{"sku":"0100410BBID075","ean":"5415015438291","quantity":0},{"sku":"0100410BBID080","ean":"5415015438338","quantity":0},{"sku":"0100410BBID085","ean":"5415015438376","quantity":0},{"sku":"0100410BBID090","ean":"5415015438413","quantity":0},{"sku":"0100410BBIE070","ean":"5415015438260","quantity":0},{"sku":"0100410BBIE075","ean":"5415015438307","quantity":0},{"sku":"0100410BBIE080","ean":"5415015438345","quantity":0},{"sku":"0100410BBIE085","ean":"5415015438383","quantity":0},{"sku":"0100410BBIE090","ean":"5415015438420","quantity":0},{"sku":"0100410BDMB070","ean":"5410760934043","quantity":0},{"sku":"0100410BDMB075","ean":"5410760934081","quantity":0},{"sku":"0100410BDMB080","ean":"5410760934128","quantity":0},{"sku":"0100410BDMB085","ean":"5410760934166","quantity":0},{"sku":"0100410BDMB090","ean":"5410760934203","quantity":0},{"sku":"0100410BDMC070","ean":"5410760934050","quantity":0},{"sku":"0100410BDMC075","ean":"5410760934098","quantity":0},{"sku":"0100410BDMC080","ean":"5410760934135","quantity":0},{"sku":"0100410BDMC085","ean":"5410760934173","quantity":0},{"sku":"0100410BDMC090","ean":"5410760934210","quantity":0},{"sku":"0100410BDMD070","ean":"5410760934067","quantity":0},{"sku":"0100410BDMD075","ean":"5410760934104","quantity":0},{"sku":"0100410BDMD080","ean":"5410760934142","quantity":0},{"sku":"0100410BDMD085","ean":"5410760934180","quantity":0},{"sku":"0100410BDMD090","ean":"5410760934227","quantity":0},{"sku":"0100410BDME070","ean":"5410760934074","quantity":0},{"sku":"0100410BDME075","ean":"5410760934111","quantity":0},{"sku":"0100410BDME080","ean":"5410760934159","quantity":0},{"sku":"0100410BDME085","ean":"5410760934197","quantity":0},{"sku":"0100410BDME090","ean":"5410760934234","quantity":0},{"sku":"0100410BROB075","ean":"5415338872406","quantity":0},{"sku":"0100410BROB080","ean":"5415338872444","quantity":0},{"sku":"0100410BROB085","ean":"5415338872482","quantity":0},{"sku":"0100410BROB090","ean":"5415338872529","quantity":0},{"sku":"0100410BROC075","ean":"5415338872413","quantity":1},{"sku":"0100410BROC080","ean":"5415338872451","quantity":0},{"sku":"0100410BROC085","ean":"5415338872499","quantity":0},{"sku":"0100410BROC090","ean":"5415338872536","quantity":0},{"sku":"0100410BROD070","ean":"5415338872383","quantity":0},{"sku":"0100410BROD075","ean":"5415338872420","quantity":0},{"sku":"0100410BROD080","ean":"5415338872468","quantity":0},{"sku":"0100410BROD085","ean":"5415338872505","quantity":0},{"sku":"0100410BROD090","ean":"5415338872543","quantity":0},{"sku":"0100410BROE070","ean":"5415338872390","quantity":0},{"sku":"0100410BROE075","ean":"5415338872437","quantity":0},{"sku":"0100410BROE080","ean":"5415338872475","quantity":0},{"sku":"0100410BROE085","ean":"5415338872512","quantity":0},{"sku":"0100410BROE090","ean":"5415338872550","quantity":0},{"sku":"0100410CALB070","ean":"5414787841018","quantity":0},{"sku":"0100410CALB075","ean":"5414787841056","quantity":5},{"sku":"0100410CALB080","ean":"5414787841094","quantity":5},{"sku":"0100410CALB085","ean":"5414787841131","quantity":5},{"sku":"0100410CALB090","ean":"5414787841179","quantity":5},{"sku":"0100410CALC070","ean":"5414787841025","quantity":0},{"sku":"0100410CALC075","ean":"5414787841063","quantity":5},{"sku":"0100410CALC080","ean":"5414787841100","quantity":5},{"sku":"0100410CALC085","ean":"5414787841148","quantity":5},{"sku":"0100410CALC090","ean":"5414787841186","quantity":5},{"sku":"0100410CALD070","ean":"5414787841032","quantity":5},{"sku":"0100410CALD075","ean":"5414787841070","quantity":5},{"sku":"0100410CALD080","ean":"5414787841117","quantity":5},{"sku":"0100410CALD085","ean":"5414787841155","quantity":5},{"sku":"0100410CALD090","ean":"5414787841193","quantity":5},{"sku":"0100410CALE070","ean":"5414787841049","quantity":5},{"sku":"0100410CALE075","ean":"5414787841087","quantity":5},{"sku":"0100410CALE080","ean":"5414787841124","quantity":5},{"sku":"0100410CALE085","ean":"5414787841162","quantity":5},{"sku":"0100410CALE090","ean":"5414787841209","quantity":5},{"sku":"0100410COGB070","ean":"5410760893548","quantity":0},{"sku":"0100410COGB075","ean":"5410760893586","quantity":0},{"sku":"0100410COGB080","ean":"5410760893623","quantity":0},{"sku":"0100410COGB085","ean":"5410760893661","quantity":0},{"sku":"0100410COGB090","ean":"5410760893708","quantity":0},{"sku":"0100410COGC070","ean":"5410760893555","quantity":0},{"sku":"0100410COGC075","ean":"5410760893593","quantity":0},{"sku":"0100410COGC080","ean":"5410760893630","quantity":0},{"sku":"0100410COGC085","ean":"5410760893678","quantity":0},{"sku":"0100410COGC090","ean":"5410760893715","quantity":0},{"sku":"0100410COGD070","ean":"5410760893562","quantity":0},{"sku":"0100410COGD075","ean":"5410760893609","quantity":0},{"sku":"0100410COGD080","ean":"5410760893647","quantity":0},{"sku":"0100410COGD085","ean":"5410760893685","quantity":0},{"sku":"0100410COGD090","ean":"5410760893722","quantity":0},{"sku":"0100410COGE070","ean":"5410760893579","quantity":0},{"sku":"0100410COGE075","ean":"5410760893616","quantity":0},{"sku":"0100410COGE080","ean":"5410760893654","quantity":0},{"sku":"0100410COGE085","ean":"5410760893692","quantity":0},{"sku":"0100410COGE090","ean":"5410760893739","quantity":0},{"sku":"0100410CRBB070","ean":"5415015683684","quantity":0},{"sku":"0100410CRBB075","ean":"5415015683721","quantity":0},{"sku":"0100410CRBB080","ean":"5415015683769","quantity":0},{"sku":"0100410CRBB085","ean":"5415015683806","quantity":0},{"sku":"0100410CRBB090","ean":"5415015683844","quantity":0},{"sku":"0100410CRBC070","ean":"5415015683691","quantity":0},{"sku":"0100410CRBC075","ean":"5415015683738","quantity":0},{"sku":"0100410CRBC080","ean":"5415015683776","quantity":0},{"sku":"0100410CRBC085","ean":"5415015683813","quantity":0},{"sku":"0100410CRBC090","ean":"5415015683851","quantity":0},{"sku":"0100410CRBD070","ean":"5415015683707","quantity":0},{"sku":"0100410CRBD075","ean":"5415015683745","quantity":0},{"sku":"0100410CRBD080","ean":"5415015683783","quantity":0}],"totalElements":282582,"size":100,"number":1}'

Then I use

data=json.loads(rawdata)
df_data = pd.json_normalize(data, record_path =['content'])
df_data

but this code only reads the data of the last page ('number':1). How can I get all the 'content' data of all pagenumbers in one df.

user_843953 · Answer

You overwrite the old rawdata every time you get a new page. After the loop, rawdata contains the contents of the last iteration. You need to process each rawdata as you obtain it, and append the value of every new content key to a list. Then create a dataframe out of that list.

all_content = []
for page_num in range(2):
    ...
    rawdata = response.read()
    data = json.loads(rawdata)
    all_content.extend(data["content"])

At this point, all_content contains all the values from all the pages. Now you can create your dataframe:

df_data = pd.DataFrame(all_content)

json.loads only reads the data of the last page in Python

1 Answers