Limited API - call all results from it with do while loop

Question

Working with this EIA API with a limit of 5000 results at once. What I'm trying to do is to loop and when I reach 5000 results loop again until all results are fetched and put in an array.

This is what I have done so far but they are still 5000 on the page. Not the actual number since they are around 10k.

In any case I don't know how much will be length when I call it but it will be more than 5000 for sure.

$offset = 0; 
$length = 5000;
      
$allData = array();

$url = "https://api.eia.gov/v2/international/data/v2/data?api_key=xxxxxxxxxx&start=1960&sort[0][direction]=desc&offset=$offset&length=$length";

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);  
                
do {
    curl_setopt($ch, CURLOPT_URL, $url);

    $jsonData = curl_exec($ch);
    $response = json_decode($jsonData);
                    
    foreach($response->response->data as $row1){
        $urlData[] = $row1;
    }                
    $received = $response->total; // total number of results

    $offset = $received - $length;
    $allData = array_merge($allData, $urlData);  
    //$offset += $received;
                
} while ( $offset == $received );

curl_close($ch);

Update: result

stdClass Object
(
    [response] => stdClass Object
        (       
            [count query execution] => 0.029705004
            [total] => 9517
            [data]
               (
                      .....
               )
)

So the total should be $response->response->total but when I add it to the while ( $response->response->total > 0 ); it timed-out

UPDATE 2: var_dump($response)

object(stdClass)#5003 (3) {
  ["response"]=>
  object(stdClass)#5002 (7) {
    float(2.6497E-5)
    ["count query execution"]=>
    float(0.037652072)
    ["total"]=>
    int(9517)
    ["data"]=>
    array(0) {
    }
  }
  ["request"]=>
      ["sort"]=>
      array(1) {
        [0]=>
        object(stdClass)#1 (1) {
          ["direction"]=>
          string(4) "desc"
        }
      }
      ["offset"]=>
      int(9517)
      ["length"]=>
      int(5000)
    }
  }
  ["apiVersion"]=>
  string(5) "2.0.2"
}

for some reason with this setup ["data"]=> array(0) () is empty but should not be. This is actual what is (should return) returning

Array
(
    [0] => stdClass Object
        (
            [count query execution] => 0.039096933
            [total] => 9517
            [data] => Array
                (
                    [0] => stdClass Object
                        (
                            [year] => 2021
                            [Id] => 57
                            [Name] => name
                        )

                    [1] => stdClass Object
                        (
                            [year] => 2021
                            [Id] => 57
                            [Name] => name
                        )
                            ..........
                    [4999] => stdClass Object
                        (
                            [year] => 2021
                            [Id] => 57
                            [Name] => name
                        )
        )      
)

user_4964822 · Answer

Just creating a URL template once doesn't update the variable values on it's changes. You will have to set them each time you update them.
You will also have to loop till the total number of records fetched for a particular offset is 0 for the pagination limit of 5000 (which is your $length). See the below snippet with the changes needed written inside the comments.

Snippet:

response->data as $row1){
        $allData[] = $row1; #Change-2 
    }

    $offset += count($response->response->data); // #Change-3

} while ( count($response->response->data) > 0 ); #Change-4

curl_close($ch);

Limited API - call all results from it with do while loop

1 Answers