I have a project currently running Laravel 5.7.
I have a bunch of IoT sensors that are sending data into Google Datastore.
I am wanting to query this data so that I can show a graph of this data on the frontend of my app.
The frontend makes a request to my project, my project makes the request to Google Datastore, and then builds up the result to return it as JSON to the frontend.
There's a fair bit of data (currently 1200 rows), and it takes about 17 seconds to fetch and render on the screen. Which is longer than I would like.
Here is my current code to fetch this IoT Data:
public function fetchData()
{
dump("GDS query starting " . date("Y-m-d H:i:s"));
$query = $this->datastore->query()
->kind('IotEvent')
->filter('device_id', '=', 'abc123')
->filter('published_at', '>=', '2018-09-19T04:52:01.429Z')
->order('published_at', Query::ORDER_ASCENDING)
->projection(['current_temperature', 'target_temperature', 'published_at']);
$results = $this->datastore->runQuery($query);
dump("GDS query finished " . date("Y-m-d H:i:s"));
return($this->transformData($results));
}
private function transformData($results)
{
dump("GDS transform starting " . date("Y-m-d H:i:s"));
$data = [];
foreach ($results as $result) {
array_push($data, $result->get());
}
dump("GDS transform finished " . date("Y-m-d H:i:s"));
return $data;
}
According to the dump()s the request to Datastore is done within a second. But looping over each row, getting the values, and pushing them to the array takes 14 seconds.
I can't seem to find any method on $result that will allow me to pull out all the results, and no documentation seems to hint that a method like this exists.
On this GitHub issue they hint at using iterator_to_array() which I have never heard of before. However, using it like below just returns me a bunch of empty JSON objects on my frontend:
private function transformData($results)
{
dump("GDS transform starting " . date("Y-m-d H:i:s"));
$data = iterator_to_array($results, false);
dump("GDS transform finished " . date("Y-m-d H:i:s"));
return $data;
}
Is there a method to make this process a lot quicker, or anything I can do to optimise the current process of looping each row? Once in production, each request to fetch data can be expected to return around 5000 rows, so 5 times what I am currently fetching.
I am using the official Google Datastore Library.