Is there a way to form batched requests using python requests module ? I want to send multiple homogenous http API requests in a single POST request

Rohan Parulekar 0
Viewed 23

Is there a way to form batched requests using python requests module ? I want to send multiple homogenous http API requests in a single POST request. I am trying to use the GCP documentation https://cloud.google.com/dns/docs/reference/batch?hl=en_US&_ga=2.138235123.-2126794010.1660759555 to create multiple DNS records in a single POST request. Can anyone help with a simple example of how this could be achieved using python requests module ?

This is how the sample POST request will look like:

POST /batch/farm/v1 HTTP/1.1
Authorization: Bearer your_auth_token
Host: www.googleapis.com
Content-Type: multipart/mixed; boundary=batch_foobarbaz
Content-Length: total_content_length

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:12930812@barnyard.example.com>

GET /farm/v1/animals/pony

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:12930812@barnyard.example.com>

PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"

{
  "animalName": "sheep",
  "animalAge": "5"
  "peltColor": "green",
}

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:12930812@barnyard.example.com>

GET /farm/v1/animals
If-None-Match: "etag/animals"

--batch_foobarbaz--

Basically; the main intention here is to not overload the remote API with multiple http requests causing the rate limit throttling but instead use batched http requests so that the remote API gets only a single batched request embedded with multiple requests in the form of parts.

python-requests python
2 Answers

No. HTTP doesn't work that way. You can send multiple requests simultaneously using threads, but you can't send multiple POSTs through a single request.

Tim Roberts0

According to the doc, the individual batch HTTP requests are supposed to go in the body of the request. You can try building the requests up manually. Like so:

import requests

body = """
--batch_foobarbaz
Content-Type: application/http
Content-ID: <item1:12930812@barnyard.example.com>

GET /farm/v1/animals/pony

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item2:12930812@barnyard.example.com>

PUT /farm/v1/animals/sheep
Content-Type: application/json
Content-Length: part_content_length
If-Match: "etag/sheep"

{
  "animalName": "sheep",
  "animalAge": "5"
  "peltColor": "green",
}

--batch_foobarbaz
Content-Type: application/http
Content-ID: <item3:12930812@barnyard.example.com>

GET /farm/v1/animals
If-None-Match: "etag/animals"

--batch_foobarbaz--
"""

response = requests.post(
    "https://www.googleapis.com/batch/API/VERSION/batch/form/v1",
    data=body,
    headers={
        'Authorization': 'Bearer your_auth_token',
        'Host': 'www.googleapis.com',
        'Content-Type': 'multipart/mixed; boundary=batch_foobarbaz'
    }
)

Of course, you'd have to build the individual requests manually:

Content-Type: application/http
Content-ID: <item1:12930812@barnyard.example.com>

GET /farm/v1/animals/pony

Unless you can find a library that can construct HTTP requests according to the HTTP/1.1 RFC. I can't think of one off the top of my head.

rdas0
Related