How to save JSON responses with asynchronous requests?

user_10140310 0
Viewed 2867

I have a question regarding asynchronous requests:

How do I save response.json() to a file, on the fly?

I want to make a request and save response to a .json file, without keeping it in memory.

import asyncio
import aiohttp


async def fetch(sem, session, url):
    async with sem:
        async with session.get(url) as response:
            return await response.json() # here


async def fetch_all(urls, loop):
    sem = asyncio.Semaphore(4) 
    async with aiohttp.ClientSession(loop=loop) as session:
        results = await asyncio.gather(
            *[fetch(sem, session, url) for url in urls]
        )
        return results


if __name__ == '__main__':

    urls = (
        "https://public.api.openprocurement.org/api/2.5/tenders/6a0585fcfb05471796bb2b6a1d379f9b",
        "https://public.api.openprocurement.org/api/2.5/tenders/d1c74ec8bb9143d5b49e7ef32202f51c",
        "https://public.api.openprocurement.org/api/2.5/tenders/a3ec49c5b3e847fca2a1c215a2b69f8d",
        "https://public.api.openprocurement.org/api/2.5/tenders/52d8a15c55dd4f2ca9232f40c89bfa82",
        "https://public.api.openprocurement.org/api/2.5/tenders/b3af1cc6554440acbfe1d29103fe0c6a",
        "https://public.api.openprocurement.org/api/2.5/tenders/1d1c6560baac4a968f2c82c004a35c90",
    ) 

    loop = asyncio.get_event_loop()
    data = loop.run_until_complete(fetch_all(urls, loop))
    print(data)

For now, the script just prints JSON files, and I can save them once they're all scraped:

data = loop.run_until_complete(fetch_all(urls, loop))
for i, resp in enumerate(data):
    with open(f"{i}.json", "w") as f:
        json.dump(resp, f)

But it doesn't feel right to me as it will definitely fail once I run out of memory for example.

Any suggestions?

Edit

Limited my post to only one question

aiohttp python-asyncio python
1 Answers

How do I save response.json() to a file, on the fly?

Don't use response.json() in the first place, use the streaming API instead:

async def fetch(sem, session, url):
    async with sem, session.get(url) as response:
        with open("some_file_name.json", "wb") as out:
            async for chunk in response.content.iter_chunked(4096)
                out.write(chunk)
user_16008980
Related