Detect the final page of a paginated cURL response in a Bash loop

user_7848350 0
Viewed 6202

Goal

Include all objects from a paginated REST API call in a single JSON file using cURL and Bash. This combined list will be fed into a Power BI report.

Details

A request returns a maximum of 100 objects. There are 400+ objects total. The total grows over time. I don't want to maintain a script that includes something like for set in 0 100 200 300 400 ; do because it requires me to manually match the sets to the current number of objects. To save labor, I would like a script to auto-detect when the final page has been processed and then break.

To achieve my goal, the plan I have come up with so far is to extract each incremental set of 100 items into its own JSON file, then assemble them with cat and extract relevant JSON keys/values using JQ. The reason for Bash is that it is the only programming I know.

Attempt

(Based on this question and answer)

for ((i=0; ; i+=100)); do
    contents=$(curl -u "username:password" -H "Content-Type: application/json" "https://<url>/api/core/v3/places?count=100&startIndex=$i")
    echo "$contents" > $i.json
    if [[ $contents =~ 'list" : [ ]' ]]
    then break
    fi
done

Result

All pages export as expected except the first and last pages:

  • The first startIndex should be 0 but the code makes the startIndex 100. I have tried a number of variations with i but continue to fail.
  • [edit: solved, thank you @weirdan] Neither list":null nor next":null end the loop. The script exports incremented JSON files indefinitely.~

Reference

First page of returned paginated JSON

{
  "itemsPerPage" : 100,
  "links" : {
    "next" : "https://<url>/api/core/v3/places?sort=titleAsc&count=100&startIndex=0" <--- with my script, startIndex is erroneously 100
  },
  "list" : [ {
...

Intermediate pages

{
  "itemsPerPage" : 100,
  "links" : {
    "previous" : "https://<url>/api/core/v3/places?sort=titleAsc&count=100",
    "next" : "https://<url>/api/core/v3/places?sort=titleAsc&count=100&startIndex=200"
  },
  "list" : [ {
...

Final page

{
  "itemsPerPage" : 100,
  "links" : {
    "previous" : "https://<url>/api/core/v3/places?sort=titleAsc&count=100&startIndex=400"
  },
  "list" : [ {
...

Empty page

{
  "itemsPerPage" : 100,
  "list" : [ ],
  "startIndex" : 500
}

Thank you for any advice or ideas.

jq curl api bash json
1 Answers

Assuming my theory about startIndex holds water and implementing @CharlesDuffy's suggestion about jq, this becomes

for ((i=0; ; i+=100)); do
    contents=$(curl -u "username:password" -H "Content-Type: application/json" "https://<url>/api/core/v3/places?count=100&startIndex=$i")
    echo "$contents" > $i.json
    if jq -e '.list | length == 0' >/dev/null; then 
       break
    fi <<< "$contents"
done
user_4506110
Related