Tensorflow serving failing with std::bad_alloc

userbIfh 0
Viewed 44

I'm trying to run tensorflow-serving using docker compose (served model + microservice) but the tensorflow serving container fails with the error below and then restarts.

microservice | To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
tensorflow-serving      | terminate called after throwing an instance of 'std::bad_alloc'
tensorflow-serving      |   what():  std::bad_alloc
tensorflow-serving | /usr/bin/tf_serving_entrypoint.sh: line 3: 7 Aborted 
(core dumped) tensorflow_model_server --port=8500 --rest_api_port=8501 --model_name=${MODEL_NAME} --model_base_path=${MODEL_BASE_PATH}/${MODEL_NAME} "$@"

I monitored the memory usage and it seems like there's plenty of memory. I also increased the resource limit using Docker Desktop but still get the same error. Each request to the model is fairly small as the microservice is sending tokenized text with batch size of one. Any ideas?

tensorflow-serving tensorflow docker-compose
2 Answers

I was encountering the same problem, and this fixed worked for me:

  • uninstalled and reinstalled:
    • tensorflow, tensorflow-gpu, etc to 2.9.0, (and trained and built my model)
    • docker pull and docker run tensorflow/serving:2.8.0 (this did the trick and finally got rid of this problem.)
Hanafi Haffidz0

Had the same error when using tensorflow/serving:latest. Based on Hanafi's response, I used tensorflow/serving:2.8.0 and it worked.

For reference, I used

sudo docker run -p 8501:8501 --mount type=bind,source= \
[PATH_TO_MODEL_DIRECTORY],target=/models/[MODEL_NAME] \ 
-e MODEL_NAME=[MODEL_NAME] -t tensorflow/serving:2.8.0
Bina Mircea0
Related