How can I speed up the loading of models in Keras and Tensorflow?

Question

Let's assume that I have a large amount of trained keras models and I have to sometimes load them and make predictions. I need to load every model and predict data in the fastest possible way. I think that the fastest solution is to store them in the memory however I guess that it is not good way because soon ot later the RAM will overflow? So at this point I achieve the highest performance with something like this.

K.clear_session()
random_model = load_model('path/to/' + str(random_model))
results = random_model(final_input_row)

In addition I have five models which I use almost all the time and in this case the performance is even more important. I load them during starting the server and I have a constant access to them.

graph = tf.get_default_graph()

with graph.as_default():

    constant_model = load_model(
            'path/to/constant_model')

prediction:

with graph.as_default():
    results = constant_model(final_input_row)

The problem is that during K.clear_session() which I execute during loading random_models I lost them from the memory. Without K.clear_session() loading of random_models last too long. Do you have any ideas how can I solve this? I can even use completely different methods.

UPDATE

I try to do something like this:

class LoadModel:
    def __init__(self, path):
        self.path = path
        self.sess  = tf.Session(config=config)
        self.graph = tf.get_default_graph()
        K.set_session(self.sess)
        with self.graph.as_default():
            self.model = load_model(self.path)

    def do_predictions(self, x):
        with self.graph.as_default():
            return self.model.predict(x)

And then when I execute:

random_model = LoadModel('./path/to/random_model.h5')
results = random_model.do_predictions(final_input_row)

It takes something about 3 seconds to predict data. In the case of random_models when I have a lot of models it is acceptable. However in the case of constant_models when I have five of them and I need to have constant access to them it lasts too long. Until now, I did it in such a way that I was loading the model during starting the Django server and I stored this in the memory and then I could just run results = constant_model.do_predictions(final_input_row) and it was very fast. It works okay until I run random_models, after that in every request I get

tensorflow.python.framework.errors_impl.InvalidArgumentError: Tensor lstm_14_input:0, specified in either feed_devices or fetch_devices was not found in the Graph
[24/Jun/2019 10:11:00] "POST /request/ HTTP/1.1" 500 17326
[24/Jun/2019 10:11:02] "GET /model/ HTTP/1.1" 201 471
Exception ignored in: >
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1455, in __del__
    self._session._session, self._handle, status)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: No such callable handle: 140576717540032
[24/Jun/2019 10:11:02] "GET /model/ HTTP/1.1" 201 471
[24/Jun/2019 10:11:07] "GET /model/ HTTP/1.1" 201 471

Obviously it works properly if I run constant_model = LoadModel('./path/to/constant_model.h5') before every results = constant_model.do_predictions(final_input_row) however as I have mentioned it is too slow. Any ideas how to solve this?

UPDATE2

I try in this way

session  = tf.Session(config=config)
K.set_session(session)

class LoadModel:
    def __init__(self, path):
        self.path = path

        self.graph = tf.get_default_graph()
        with self.graph.as_default():
            self.model = load_model(self.path)

    def do_predictions(self, x):
        with self.graph.as_default():
            return self.model.predict(x)

but still I get TypeError: Cannot interpret feed_dict key as Tensor: Tensor Tensor("Placeholder:0", shape=(1, 128), dtype=float32) is not an element of this graph.

SOLUTION Below is my working solution. If anyone knows an even more efficient way of loading the model that meets the requirements described above, I will be grateful for the message.

class LoadModel:

    def __init__(self, model_path):
        with K.get_session().graph.as_default():
            self.model = load_model(model_path)
            self.model._make_predict_function()
            self.graph = tf.get_default_graph()

    def predict_data(self, data):
        with self.graph.as_default():
            output = self.model.predict(data)
            return output

user_8382707 · Answer

As you said that you are using all the models all the time, it will be good to keep them in memory so that you are not loading and predicting every time you make a request. For example, you should define a class that loads a model and keep different instances of this class for different models. This is not a tested code, so you may have to make some changes.

# if you are using GPU
config = tf.ConfigProto()
config.gpu_options.allow_growth = True

class LoadModel:
    def __init__(self, path_to_model):
        self.path  = path
        self.sess  = tf.Session(config=config)
        self.graph = tf.get_default_graph()
        K.set_session(self.sess) 
        self.model = self.load()

    def load(self):
        with graph.as_default():
            model = load_model(self.path)
        return model

    def do_predictions(self, x):
        return self.model.predict(x)

How can I speed up the loading of models in Keras and Tensorflow?

1 Answers