aiohttp check request headers after receiving response

user_17552215 0
Viewed 71

here's the problem: I have a list of authorization headers to check. For that, I use aiohhtp

def make_tasks(session, proxies, unchecked_headers):
    tasks = list()
    for unchecked_header in unchecked_headers:
        current_proxy = proxies.pop()
        headers['authorization'] = unchecked_header
        t = session.get(url, proxy=current_proxy, headers=headers)
        tasks.append(t)
    return tasks


async def check_headers(proxies, unchecked_headers):
    async with aiohttp.ClientSession() as s:
        tasks = make_tasks(s, proxies, unchecked_headers)
        results = await asyncio.gather(*tasks)
        for result in results:
            ...

Now, depending on the response code, I need to log some information about that authorization header. The issue is that no where in the response it mentions the actual header. So, I get all of the responses back, yet I don't know which response corresponds to which header, as they are asynchronous.

I looked around, and didn't find a way to check which headers were sent initially on the ClientResponse object. What can I do here?

python-requests aiohttp header python
1 Answers

You can use asyncio.as_completed + wrapper around session.get to return response and any additional data you want. For example:

import aiohttp
import asyncio


url = "https://httpbin.org/get"

unchecked_headers = [
    "A",
    "B",
    "C",
]
proxies = ["Proxy1", "Proxy2", "Proxy3"]
headers = {}


def make_tasks(session, proxies, unchecked_headers):
    async def _wrapper(t, *args):
        response = await t
        return response, *args

    tasks = list()
    for unchecked_header in unchecked_headers:
        current_proxy = proxies.pop()
        headers["authorization"] = unchecked_header

        # I commented this out because I don't have access to proxy:
        # t = session.get(url, proxy=current_proxy, headers=headers)

        t = session.get(url, headers=headers)
        tasks.append(_wrapper(t, current_proxy, unchecked_header))
    return tasks


async def check_headers(proxies, unchecked_headers):
    async with aiohttp.ClientSession() as s:
        for task in asyncio.as_completed(
            make_tasks(s, proxies, unchecked_headers)
        ):
            response, proxy, header = await task
            print(response.url, proxy, header)


async def main():
    await check_headers(proxies, unchecked_headers)


if __name__ == "__main__":
    asyncio.run(main())

Prints:

https://httpbin.org/get Proxy3 A
https://httpbin.org/get Proxy1 C
https://httpbin.org/get Proxy2 B
user_100359850
Related