How long in downtime of Vertical Pod Autoscaling to create a new pod with the optimized resources?

user_11506883 0
Viewed 390

I am new to K8s autoscaling. I have a stateful application I am trying to find out which autoscaling method works for me. According to the documentation:

if pods don't have the correct resources set, the Updater component of VPA kills them so that they can be recreated by their controllers with the updated requests.

I want to know the downtime for kills the existing pod and creating the new ones. Or at least how can I measure it for my application?

I am comparing the HPA and VPA approaches for my application.

the follow-up question is - how long does it take in HPA to create a new pod in scaling up?

horizontal-scaling horizontal-pod-autoscaling autoscaling kubernetes-pod kubernetes
1 Answers

There are few things to clear out here:

  • VPA does not create nodes, Cluster Autoscaler is used for that. Vertical Pod Autoscaler allocates more (or less) CPUs and memory to existing pods and CA scales your node clusters based on the number of pending pods.

  • Whether to use HPA, VPA, CA, or some combination, depends on the needs of your application. Experimentation is the most reliable way to find which option works best for you, so it might take a few tries to find the right setup. HPA and VPA depend on metrics and some historic data. CA is recommended if you have a good understanding of your pods and containers needs.

  • HPA and VPA should not be used together to evaluate CPU/Memory. However, VPA can be used to evaluate CPU or Memory whereas HPA can be used to evaluate external metrics (like the number of HTTP requests or the number of active users, etc). Also, you can use VPA together with CA.

  • It's hard to evaluate the exact time needed for VPA to adjust and restart pods as well as for HPA to scale up. The difference between best case scenario and worse case one relies on many factors and can make a significant gap in time. You need to rely on metrics and observations in order to evaluate that.

  • Kubernetes Metrics Server collects resource metrics from Kubelets and exposes them in Kubernetes apiserver through Metrics API for use by Horizontal Pod Autoscaler and Vertical Pod Autoscaler.

Below are some useful sources that would help you understand and choose the right solution for you:

EDIT:

Scaling up is a time sensitive operation. You should consider the average time it can take your pods to scale up. Two example scenarios:

  1. Best case scenario  - 4 minutes:
  • 30 seconds : Target metrics values updated: 30-60 seconds
  • 30 seconds : HPA checks on metrics values: 30 seconds
  • < 2 seconds:  pods created and goes into pending state  - 1 second
  • < 2 seconds : CA sees the pending pods and fires up the calls to provision nodes  - 1 second
  • 3 minutes:  Cloud provider provision the nodes & K8 waits for them till they are ready: up to 10 minutes (depends on multiple factors)
  1. (Reasonable) Worst case scenario  - 12 minutes:
  • 60 seconds : Target metrics values updated
  • 30 seconds : HPA checks on metrics values
  • < 2 seconds : pods created and goes into pending state
  • < 2 seconds : CA sees the pending pods and fires up the calls to provision nodes
  • 10 minutes : Cloud provider provision the nodes & K8 waits for them till they are ready minutes (depends on multiple factors, such provider latency, OS latency, boot strapping tools, etc.)

Again, it is hard to estimate the exact time it would take so observation and metrics are the key here.

user_115608780
Related