How to speed up the Keras model.predict?

user_10934417 0
Viewed 3383

I have trained a LSTM model and tried to make predictions for all test observations. However, it takes forever for keras model.predict to calculate all predictions. Is there a way to speed up this process? Assume there are two features (x1 & x2) for each prediction. The length of each feature (x1 & x2) are 33. Such as [32,1,17,.......,0]. I need to make 1M predictions. My code is

predictions = np.argmax(make.predict([x1, x2]), axis = -1)

Any idea to speed up? Many Thanks

data-science keras tensorflow python
2 Answers

Actually, Keras model is a main architecture to perform, training, retraining, finetuning and summary and model wise changes, While doing predictions and deployment, we need to use frozen inference graph of keras model.

We should also gone for Frozen graph optimization with use of TensorRT, OpenVINO and many other Model Optimization techniques.

Here i have added snippet to convert graph from Keras model

Links to freeze graph

Convert Keras Model to TensorFlow frozen graph

Save, Load and Inference From TensorFlow 2.x Frozen Graph

It might looks lengthy, but its actual method

#Freeze Graph

# Convert Keras model to ConcreteFunction
full_model = tf.function(lambda x: model(x))
full_model = full_model.get_concrete_function(
    tf.TensorSpec(model.inputs[0].shape, model.inputs[0].dtype))

# Get frozen ConcreteFunction
frozen_func = convert_variables_to_constants_v2(full_model)
frozen_func.graph.as_graph_def()


# Save frozen graph from frozen ConcreteFunction to hard drive
tf.io.write_graph(graph_or_graph_def=frozen_func.graph,
                  logdir="./frozen_models",
                  name="frozen_graph.pb",
                  as_text=False)

#Load Frozen Graph

   # Load frozen graph using TensorFlow 1.x functions
with tf.io.gfile.GFile("./frozen_models/frozen_graph.pb", "rb") as f:
    graph_def = tf.compat.v1.GraphDef()
    loaded = graph_def.ParseFromString(f.read())

# Wrap frozen graph to ConcreteFunctions
frozen_func = wrap_frozen_graph(graph_def=graph_def,
                                inputs=["x:0"],
                                outputs=["Identity:0"],
                                print_graph=True)
user_82080060

Try to do the predictions in parallel, like 20-40 threads simultaneously using multiprocessing:

import multiprocessing

output=[]
data = range(0,len(x1.shape[0])) #length of x1 set, not the 33 columns

def f(x):

    predictions = np.argmax(model.predict([x1, x2]), axis = -1)  
    return predictions

def mp_handler():
    p = multiprocessing.Pool(40) # number of parallel threads
    r=p.map(f, data)
    return r
    
if __name__ == '__main__':
    output.append(mp_handler())

You will get all your predictions at output[0]

user_69016900
Related