I've been deploying some custom trained models to Vertex AI, but lately, the feature to auto-scale has not been working properly on the later endpoints. Basically, despite the traffic, the endpoint doesn't auto-scale.
I have an older endpoint which works as intended, so I deployed the same model to a different endpoint with the same configuration (same machine specs, same GPU, min 1 machine, max 3 machines, 60% threshold to auto-scale), created it's own task queue and then proceeded to send the same requests to both endpoints at the same time.
The older endpoint worked as intended, scaling up and down depending on the incoming traffic. The newer one, on the other hand, stayed stuck at one machine the entire time.
I can force it to scale up if I lower the threshold to 15-20%, and it does scale up as the requests come in. However, it does not scale down once it has finished processing the requests and it stays with all the machines on even when there has not been any traffic for hours.
So, what may be preventing the newer endpoint to scale up as the traffic increases, given that the older endpoint does scale up and down as intended with the same traffic? And perhaps more importantly, what prevents it to scale down if I force it to scale up?