Bash loop a curl request, output to file and stop until empty response

As stated in author's comment on his/her own post, the returned data is in json format. The author didn't ask how to append two json files, but it is a necessary step for him/her to accomplish his/her job. In order to append two json's, json1 and json2, maybe skipping json1 last byte } and json2 first byte {, and appending , between them would be enough. Here I am using jq to join two jsons as a more generic approach.

In the examples shown below, the nextjsonchunk file is the json file got at each request. If it has contents, it is appended to the mainjsonfile with jq. If it seems to be empty (inferred by its size) the loop breaks and the result is moved to the present folder and cleanup is made.

Using curl:

#!/usr/bin/env bash

tempfolder=/dev/shm  # temporary memory parition, avaiable in ubuntu
emptyjsonize=10      # the minimum json file length, to be used as a threshold

for ((counter=1; 1; counter++))
do
  curl "https://api.io/v1/candidates?page=${counter}&per_page=500" \
    --header "Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
    --ouput $tempfolder/nextjsonchunk
  if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
  jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result

Alternativelly, using wget:

#!/usr/bin/env bash

tempfolder=/dev/shm  # temporary memory parition, avaiable in ubuntu
emptyjsonize=10      # the minimum json file length, to be used as a threshold

for ((counter=1; 1; counter++))
do
  wget "https://api.io/v1/candidates?page=${counter}&per_page=500" \
    --header="Authorization: Basic aklsjdl;fakj;l;kasdflkaj" \
    --ouput-document $tempfolder/nextjsonchunk
  if [ $(wc -c <$tempfolder/nextjsonchunk) -le $emptyjsonize ]; then break; fi
  jq -s '.[0]*.[1]' $tempfolder/mainjsonfile $tempfolder/nextjsonchunk > $folder/mainjsonfile
done
rm $tempfolder/nextjsonchunk # cleaning up
mv $tempfolder/mainjsonfile ./jsonresultfile # end result

• It is a good idea to take two sample json and test the merging between them, to check if it is being done properly.

• It is also good to assure if the empty json file check is ok. The 10 byte was just a guess.

• A tmpfs (in memory) partition, /dev/shm was used in the examples, to avoid a many writes, but its use is optional.

You can use break to end the loop at any point:

#!/bin/bash
for ((counter=1; 1; counter++)); do
  curl -o gettext.txt --request GET \
    --url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
    --header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj'
  if [ ! -s gettext.txt ]; then
    break;
  fi
  # do something with gettext.txt
  # as in your question, it will be overwritten in the next iteration
done
echo "$counter"
echo "All done"

Like this?

#!/bin/bash

# Basic while loop
counter=1
while true; do
    data=$(curl --request GET \
        --url "https://api.io/v1/candidates?page=${counter}&per_page=500" \
        --header 'Authorization: Basic aklsjdl;fakj;l;kasdflkaj')
    [[    $data ]] || break
    echo "$data"   >> gettext.txt
    ((counter++))
done

echo $counter
echo All done