Requests/aiohttp: closing response objects

user_7954504 0
Viewed 2334

I'm a bit confused about the need to .close() a response object in both requests and aiohttp. (Note that this is a separate instance method than session.close()--I'm talking about the response object itself.)

  • Does Response (requests) or ClientResponse (aiohttp) ever need explicitly call .close()?
  • If not, what is the purpose of using the response itself as a context manager? (async with session.request('GET', 'https://www.pastebin.com') below.) Why define the two dunder methods for this if it gets closed implicitly as shown below?

Some simple tests (below) seem to imply that responses are closed automatically when they are defined inside of a Session context manager. (Which itself calls self.close() in __exit__ or __aexit__. But this is the closing of the Session, not the Response object.)

Example - requests

>>> import requests
>>> 
>>> with requests.Session() as s:
...     resp = s.request('GET', 'https://www.pastebin.com')
...     resp.raise_for_status()
...     print(resp.raw.closed)  # `raw` is urllib3.response.HTTPResponse object
...     print(resp.raw._pool)
...     print(resp.raw._connection)
...     c = resp.text
... 
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None
>>>
>>> while 1:
...     print(resp.raw.closed)
...     print(resp.raw._pool)
...     print(resp.raw._connection)
...     break
... 
True
HTTPSConnectionPool(host='pastebin.com', port=443)
None

Example - aiohttp

>>> import asyncio
>>> import aiohttp
>>>
>>> async def get():
...     async with aiohttp.ClientSession() as s:
...         # The response is already closed after this `with` block.
...         # Why would it need to be used as a context manager?
...         resp = await s.request('GET', 'https://www.pastebin.com')
...         print(resp._closed)
...         print(resp._connection)
...         print(resp._released)
...         c = await resp.text()
...     print()
...     print(resp._closed)
...     print(resp._connection)
...     print(resp._released)
...     return c
... 
>>> c = asyncio.run(get())  # Python 3.7 +
False
Connection<ConnectionKey(host='pastebin.com', port=443, is_ssl=True, ssl=None, proxy=None, proxy_auth=None, proxy_headers_hash=None)>
False

True
None
False

Here's the source for requests.models.Response. What does "Should not normally need to be called explicitly" mean? What are the exceptions?

def close(self):
    """Releases the connection back to the pool. Once this method has been
    called the underlying ``raw`` object must not be accessed again.
    *Note: Should not normally need to be called explicitly.*
    """
    if not self._content_consumed:
        self.raw.close()

    release_conn = getattr(self.raw, 'release_conn', None)
    if release_conn is not None:
        release_conn()
python-requests python-3.7 aiohttp python
1 Answers

Requests: You need not explicitly call close(). request will automatically close after finished because it bases on urlopen (this is why resp.raw.closed is True), This is the simplified code after i watched session.py and adapters.py:

from urllib3 import PoolManager
import time
manager = PoolManager(10)
conn = manager.connection_from_host('host1.example.com')
conn2 = manager.connection_from_host('host2.example.com')
res = conn.urlopen(url="http://host1.example.com/",method="get")
print(len(manager.pools))
manager.clear()
print(len(manager.pools))
print(res.closed)

#2
#0
#True

Then what did the __exit__ do? It uses to clear PoolManager(self.poolmanager=PoolManager(...)) and proxy.

# session.py
def __exit__(self, *args): #line 423
    self.close()
def close(self): #line 733
    for v in self.adapters.values():
        v.close()

# adapters.py
# v.close()
def close(self): #line 307
        self.poolmanager.clear()
        for proxy in self.proxy_manager.values():
            proxy.clear()

So when should you need to use close() , as the note said Releases the connection back to the pool, because DEFAULT_POOLSIZE = 10(http/https are independent). That means if you want to access more than 10 website with one session , you can chose to close some you do not need otherwise manager will close connection from the first to the newest when you have one more. But actually you need not to care about this , you can specify pool size and it would not waste much time to rebuild connection

aiohttp aiohttp.ClientSession() is using one TCPConnector for all requests. When it triggered __aexit__ , self._connector will be closed.

Edit: s.request() is set up a connection from host but it did not get response. await resp.text() can only be done after got response, if you did not do such step(wait for response), you will exit without having response.

if connector is None: #line 132
    connector = TCPConnector(loop=loop)
...
self._connector = connector #line 151
# connection timeout
try:
    with CeilTimeout(real_timeout.connect,loop=self._loop):
    assert self._connector is not None
    conn = await self._connector.connect(
        req,
        traces=traces,
        timeout=real_timeout
        )
...
async def close(self) -> None:
        if not self.closed:
            if self._connector is not None and self._connector_owner:
                self._connector.close()
            self._connector = None
...
async def __aexit__(self,
                       ...) -> None:
        await self.close()

This is code to show what i said

import asyncio
import aiohttp
import time

async def get():
    async with aiohttp.ClientSession() as s:
        # The response is already closed after this `with` block.
        # Why would it need to be used as a context manager?
        resp = await s.request('GET', 'https://www.stackoverflow.com')
        resp2 = await s.request('GET', 'https://www.github.com')
        print("resp:",resp._closed)
        print("resp:",resp._connection)
        print("resp2:",resp2._closed)
        print("resp2:",resp2._connection)
        s.close()
        print(s.closed)
        c = await resp.text()
        d = await resp2.text()

    print()
    print(s._connector)
    print("resp:",resp._closed)
    print("resp:",resp._connection)
    print("resp2:",resp2._closed)
    print("resp2:",resp2._connection)

loop = asyncio.get_event_loop()
loop.run_until_complete(get())  # Python 3.5 +

#dead loop
user_93806940
Related