Python aiohttp returns a different reponse than python requests. I need help understanding why

user_15823444 0
Viewed 508

after the whole evening yesterday and morning today i really an help for understanding why, aiohttp request returns differently than requests request.

import requests

reqUrl = "https://api-mainnet.magiceden.io/all_collections_with_escrow_data"

headersList = {
 "Accept": "*/*",
 " User-Agent" : " Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36" 
}

payload = ""

response = requests.request("GET", reqUrl, data=payload,  headers=headersList)

print(response.text)

returns whole content {"collections":[{"symbol"....

import aiohttp
import asyncio 

headersList = {
            'authority': 'api-mainnet.magiceden.io',
            'Accept': 'application/json, text/plain, */*',
            'accept-language': 'en-US,en;q=0.9',
            'origin': 'https://magiceden.io',
            'referer': 'https://magiceden.io/',
            'sec-fetch-dest': 'empty',
            'sec-fetch-mode': 'cors',
            'sec-fetch-site': 'same-site',
            'sec-gpc': '1',
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.61 Safari/537.36',
        }

payload = ""

async def test():
    async with aiohttp.ClientSession() as session:
        get_image_data = await session.get(
                            'https://api-mainnet.magiceden.io/all_collections_with_escrow_data', headers=headersList)
        result = get_image_data.text

        print(result)
    
if __name__ == '__main__':
    asyncio.run(test())

returns:

<bound method ClientResponse.text of <ClientResponse(https://api-mainnet.magiceden.io/all_collections_with_escrow_data) [200 OK]>
<CIMultiDictProxy('Date': 'Thu, 02 Jun 2022 12:43:22 GMT', 'Content-Type': 'application/json; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'X-Powered-By': 'Express', 'X-RateLimit-Limit': '120', 'X-RateLimit-Remaining': '119', 'X-RateLimit-Reset': '1654173863', 'Access-Control-Allow-Origin': 'https://magiceden.io', 'Vary': 'Origin', 'Access-Control-Allow-Credentials': 'true', 'Cache-Control': 'public, max-age=300, s-maxage=300', 'CDN-Cache-Control': 'public, max-age=300, s-maxage=300', 'Etag': 'W/"4f27d7-Cndhwdfejd0aSIGFdSQriuQfbvE"', 'Set-Cookie': 'connect.sid=s%3AcUUsXzow-3-5kuLPJcNNndd5zVxtCIvc.ggQdFm%2FooB%2FpWho%2FqYiVWJQa4vCtQ9VZGRisUqFXigw; Domain=magiceden.io; Path=/; Expires=Thu, 02 Jun 2022 12:53:22 GMT; HttpOnly', 'X-Kong-Upstream-Latency': '242', 'X-Kong-Proxy-Latency': '0', 'Via': 'kong/2.7.2', 'CF-Cache-Status': 'DYNAMIC', 'Expect-CT': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"', 'Set-Cookie': '__cf_bm=i5THtMhPjqGPXy8zFyXSP43DLHQESpOLbWff7n_W6qE-1654173802-0-AQkuHSP7Sv+YRRSr1wmUBDKb5EOjcAiPVXyx7lvqe0NF2wmLHcFK9JFLPVkiuTpMOjp/wpiMpQU377nAimriGP0=; path=/; expires=Thu, 02-Jun-22 13:13:22 GMT; domain=.magiceden.io; HttpOnly; Secure; SameSite=None', 'Server': 'cloudflare', 'CF-RAY': '715046367a9b0219-ZRH', 'Content-Encoding': 'gzip', 'alt-svc': 'h3=":443"; ma=86400, h3-29=":443"; ma=86400')>
>

can someone help me understanding why? Thanks a lot guys...

python-requests aiohttp screen-scraping python
2 Answers

The output you see is because text is supposed to be called as an (async) method, not looked up as an attribute.

If you just want the full response body as a string, you need to await response.text() as shown in the official aiohttp tutorial:

async def test():
    async with aiohttp.ClientSession() as session:
        async with session.get(url, headers=headers_list) as response:
            text = await response.text()

    print(text)

One difference between requests.get and aiohttp.ClientSession.get is that requests fetches the whole body of the response at once and remembers it, but aiohttp doesn't. aiohttp lets you ignore the body, or read it in chunks, or read it after looking at the headers/status code.

That's why you need to do a second await: aiohttp needs to do more I/O to get the response body.

user_102957290

The text is actually in a StreamReader object in the content attribute

get_image_data = await session.get('https://api-mainnet.magiceden.io/all_collections_with_escrow_data', headers=headersList)
stream = get_image_data.content
data = await stream.read()
print(data)
user_51680110
Related