I have two classes of consumers of a service: dev-consumers and prod-consumers. They both consume a service, s, which is a StatefulSet. Scaling s is possible, but it takes around 20 hours to get an extra replica ready.
I want prod-consumers to always be prioritized over dev-consumers when using the service s. The request rate from prod-consumers is maybe around 50-100 req/s on some days, and only 1 req/s on other days. When prod-consumers is not doing a lot of requests, I want dev-consumers to get the rest of the available capacity.
This is probably a pretty common setup for many systems, so let me list some non-solutions I have found:
- Splitting
sinto dedicated services for prod and dev. This has significantly higher cost, and leaves lots of capacity unused. - I have found a token bucket-based local rate limit in istio. This could be a building block, but the token bucket would have to be modified manually.
- Similarly, the (typically built-in/default) nginx ingress controller supports rate limiting through annotations, but this is for ingresses. There might be a way to setup this for use internally in k8s. It would have the same downsides as the istio-system, but maybe without having to use a service mesh?
What I don't understand is this: If a service mesh has both rate limiting and observability, it has everything needed to do traffic prioritization. It knows how much traffic is flowing from prod-consumers to s, and if it uses token bucket based rate limiting, it could adjust these buckets based on what is observed.
Thus, I am assuming that I am missing something. Someone, somewhere, must have a solution to this where traffic can be prioritized dynamically, as all the pieces seem to be available.