Fetch multiple pages of json with jq

user_5905425 0
Viewed 1954

When calling an API, I am returned a json that can be pretty printed with 

curl "https://mywebsite.com/api/cars.json&page=1" |  jq '.'

Below is a sample output. Seeing as there are two pages of records (112 total entries and only 100 entries showing per page), what command can I use in jq to immediately fetch both pages of data?

{
  "current_page": 1,
  "per_page": 100,
  "total_entries": 112,
  "items": [
    {
      "id": 1,
      "name": "vehicleA",
      "state": "available",
      "charge": 100
    },
    {
      "id": 2,
      "name": "vehicleB",
      "state": "available",
      "charge": 75
    },
    {
      "id": 3,
      "name": "vehicleB",
      "state": "available",
      "charge": 50
    }...
jq terminal bash json
1 Answers

jq doesn't fetch pages. You need to fetch pages with multiple curl calls, and then combine the JSON outputs. For example you could get a JSON array of the combined .item properties:

url=https://mywebsite.com/api/cars.json
jq -s '.[0].items + .[1].items' <(curl "$url&page=1" | jq .) <(curl "$url&page=2" | jq .)

If you don't know in advance the number of pages you'll need, then you'll need to work a bit harder, but it's of course possible.

Get the total entries from the first page:

total=$(curl "$url&page=1" | jq .total_entries)

Compute the number of pages:

((pages = total / 100))

if ((total % 100 > 0)); then
    ((pages++))
else

And then you can write a loop to download the results page by page, generate the jq command .[0].items + ... + .[n].items and call jq with it to get the combined JSON array.

user_6419550
Related