Keras model.predict() taking unreasonable amount of time

user_15315533 0
Viewed 413

I am working on a project where we are using a compiled keras ANN-model to classify different positions based on sensor data received. These data are continuously fed to the model for it to predict via a daemon-thread collecting data in the background. We are having a problem where model.predict() takes up to 2 seconds to finish, even when entering small data-sets. The data-points are arrays containing 38 floats each. The prediction time seems unaffected by the amount of rows supplied, up to a certain amount. We have tried supplying it with only one row, and up to hundreds. The elapsed time stays around 2 seconds. Isn't this time consumption abnormally high, even for the larger data sets?

If it helps: Our program is using multi-threading to be able to collect the data from the sensors and restructure them so that they fit the predict method of the model. Two daemon threads are running in the background collecting and restructuring data, while the main thread is actively picking data from a queue of already structured data and classifying based on these. Here is the code where we classify based on the data collected:

values = []
rows = 0
while rows < 20:
    val = pred_queue.shift()
    if val != None:
        values.append(val)
        rows += 1
rows = 0
        
values = np.squeeze(values)
start_time = time.perf_counter()
predictions = model.predict(values)
elapsed_time = round(time.perf_counter() - start_time, 2)
print("Predict time: ", elapsed_time)
            
for i in range(len(predictions)):
    print(predictions[i].argmax())
    #print(f"Predicted {classification_res} in {elapsed_time}s!")

Some clarification of the code:
The shift() method returns the first entry in the pred_queue(). This will either be an array of 38 floats or None, depending on the queue being empty or not.

What could possibly make these predictions so slow?

Edit
The reason for the confusion around the prediction times is that we have run the same model on some data before compiling it. These data-points were collected from a csv file and put into a pandas dataframe and finally passed to the predict method. These data were not streamed live, but the dataset was much bigger, around 9000 rows each containing 38 floats. This prediction took 0.3 seconds when we timed it. Obviously much faster than our current speeds!

keras tensorflow machine-learning python
1 Answers

You can try to use the __call__ method directly, as the documentation of the predict method states (emphasis is mine):

Computation is done in batches. This method is designed for performance in large scale inputs. For small amount of inputs that fit in one batch, directly using __call__ is recommended for faster execution, e.g., model(x), or model(x, training=False) if you have layers such as tf.keras.layers.BatchNormalization that behaves differently during inference. Also, note the fact that test loss is not affected by regularization layers like noise and dropout.

Note that this performance hit that you are noticing could be related to the fact that the resources of the machine are limited. Investigate CPU usage, RAM usage, etc.

user_73701530
Related