I am currently experimenting with different batching configurations in Tensorflow serving and my understanding is that I can change the parameters on the fly. At the moment I am having to rebuild the container each time with updated values which is very time consuming and also not great for keeping track of what values I have set currently.
Here's an example of what I am trying to do. My current config file looks like this
max_batch_size { value: 1000 }
batch_timeout_micros { value: 5000 }
max_enqueued_batches { value: 4 }
num_batch_threads { value: 16 }
What I would like to do is edit those values on the fly (without having to rebuild the container) with python (maybe using tensorflow_serving.apis but I am not sure) and I cannot seem to find any documentation or explanation on how to do that.