I am using the requests library to put objects into an object store. The objects are fairly large (many megabytes) and performance is critical, so I'm trying to avoid copies wherever possible. Each object consists of a small header and a large blob of data. Since the default adapter for requests ultimately sits on top of http.client and http.client allows the request body to be specified as an iterable of byte-like objects, it seems very doable.
I also need a Content-Length header (the server requires it), so the generator approach listed in the documentation (using chunked encoding) won't cut it.
I have actually done it, but I had to reverse engineer it from the code using undocumented features, which always introduces a risk of a future version breaking things. I'd like to know if there is either a better solution, or if not, how fragile this is to future changes in requests. Here's my solution so far:
class Multipart:
def __init__(self, items=()):
self.items = list(items)
def __iter__(self):
return iter(self.items)
@property
def len(self):
return sum(len(item) for item in self.items)
One alternative I've thought of is to get a PreparedRequest and then override its .body (also shown in the documentation) and presumably also Content-Length. However, I'm not sure if that's going to cause other problems e.g. if the original prepared request had some form of transfer encoding pre-applied that I would need to undo in the headers. It's also a bit less convenient since that has to be done in the code that deals with the request itself (and hence knows the session, URL etc), rather than in the code that prepares the data.