TensorFlow.js speed in the browser

user_4610514 0
Viewed 3451

I've trained a simple bidirectional LSTM network in Keras (20 units) and exported the model via

tfjs.converters.save_keras_model(model, 'myModel')

The model is 53kb large. In my JavaScript app, I load the model like this

var model;
async function loadModel() {;
    model = await tf.loadModel('https://example.com/myModel.json');
}

and afterwards I run my predictions with

async function predict(input) {
    var pred = model.predict(input);
    ...
}

It takes 5-6 seconds till the model is loaded, this is fine. But what bothers me is that every call of predict() also takes 5-6 seconds. Every time. For my use case, I'd need the prediction to be extremely fast, 1 second or less.

My question is: Is this normal? Or is something wrong with my code?

Edit: Here is a codepen: https://codepen.io/anon/pen/XygXRP

BTW, model.predict is blocking the UI - how can I prevent that?

tensorflow.js tensorflow javascript performance
2 Answers

Your UI is being blocked because you have not asked the thread to await the results of the prediction, this means that it is running synchronously instead of asynchronously. You can fix this by using the await keyword e.g. var pred = await model.predict(input).

The rest of your code appears to be fine and so it looks like the delay is coming from your actual model as I saw my CPU was barely taxed to run your model.

It is worth reading the tensorflowjs blog post as they give you examples of how you can improve the efficiency of models to make faster inferences in the browser.

user_30927340

A simple thing to do: use the 'webgl' backend to get some better vector/matrix operation performance instead of 'cpu' which I believe is native Javascript and is slow AF.

user_8342140
Related