Efficiently get all results from Google Datastore and return as JSON

Viewed 375

I have a project currently running Laravel 5.7.

I have a bunch of IoT sensors that are sending data into Google Datastore.

I am wanting to query this data so that I can show a graph of this data on the frontend of my app.

The frontend makes a request to my project, my project makes the request to Google Datastore, and then builds up the result to return it as JSON to the frontend.

There's a fair bit of data (currently 1200 rows), and it takes about 17 seconds to fetch and render on the screen. Which is longer than I would like.

Here is my current code to fetch this IoT Data:

public function fetchData()
{
    dump("GDS query starting " . date("Y-m-d H:i:s"));
    $query = $this->datastore->query()
        ->kind('IotEvent')
        ->filter('device_id', '=', 'abc123')
        ->filter('published_at', '>=', '2018-09-19T04:52:01.429Z')
        ->order('published_at', Query::ORDER_ASCENDING)
        ->projection(['current_temperature', 'target_temperature', 'published_at']);

    $results = $this->datastore->runQuery($query);
    dump("GDS query finished " . date("Y-m-d H:i:s"));

    return($this->transformData($results));
}

private function transformData($results)
{
    dump("GDS transform starting " . date("Y-m-d H:i:s"));

    $data = [];

    foreach ($results as $result) {
        array_push($data, $result->get());
    }

    dump("GDS transform finished " . date("Y-m-d H:i:s"));

    return $data;
}

According to the dump()s the request to Datastore is done within a second. But looping over each row, getting the values, and pushing them to the array takes 14 seconds.

I can't seem to find any method on $result that will allow me to pull out all the results, and no documentation seems to hint that a method like this exists.

On this GitHub issue they hint at using iterator_to_array() which I have never heard of before. However, using it like below just returns me a bunch of empty JSON objects on my frontend:

private function transformData($results)
{
    dump("GDS transform starting " . date("Y-m-d H:i:s"));

    $data = iterator_to_array($results, false);

    dump("GDS transform finished " . date("Y-m-d H:i:s"));

    return $data;
}

Is there a method to make this process a lot quicker, or anything I can do to optimise the current process of looping each row? Once in production, each request to fetch data can be expected to return around 5000 rows, so 5 times what I am currently fetching.

I am using the official Google Datastore Library.

1 Answers

Are you sure the request to datastore finished within a second? Or did it just create a 'query' object and it lazy loaded when you actually tried reading values.

Regardless, what's the log term plan here? There has to be some kind of max number of rows you would return here right?

I had a similar issue that I solved by saving this json payload as a file in google cloud storage everytime something changed and then my request handler just served that file instead.

Related