requests.get(, stream=True) with Python standard library

Viewed 897

Per ps aux, it seems that import requests adds ~4mb of RAM to a code I'm trying to optimize.

The usage of requests in the respective code is pretty basic, and I read that the "regular" requests.get can be achieved via the standard library:

from urllib.request import urlopen

urlopen("www.bla.com").read()

There is however a case where verify=True, stream=True is being used.

Can this also be somehow reasonably achieved via Python (3.8) standard library?

1 Answers

If you are developing for a linux target, there may also be the possibility of calling the curl binary. I once worked on a project where we used a python lib, but ended up calling a linux tool, which was a compiled C binary and for that reason notably more performant than any python code. I can't tell about the exact memory requirements of curl, but it may be an option. It is a quite sophisticated tool able to easily perform a huge range of REST requests.

GET example downloading a website:

curl -o savedpage.html http://www.example.com/

You can run it from python like so:

import subprocess
out = subprocess.check_output(["curl", "https://www.python.org/"])
print(out)

This may be an option, but deserves comprehensive consideration, since curl may not be pre-installed on the target machine. You could also deliver the curl binary with your code, but that requires the target not to change, because you would have to pre-compile the binary.

So it is not the most straight-forward solution, on the other hand it could be a simple solution.

Related