We are using AKS version 1.19.11.
It is noticed that whenever a new rollout is in placed for our deployments or a new pod got created as part of the hpa settings or pod got restarted, We are getting high cpu usage alerts.
For example, -if a new pod got created as part of any of the above activities, will this take up more CPU than the allowed Threshold ? [ the “Maximum limit” of 1 core specified in the deployment spec and the apps are light weight and doesnt need thatmuch cpu anuyways ] ? its in turn makes sudden spike in the AzureMonitor for a short time and then it became normal.
Why the pods are taking more cpu during its startup or creation time? if the pods are not using thatmuch cpu, what will be the reason for this repeating issues?
hpa settings as below
Name: myapp
Namespace: myapp
Labels: app.kubernetes.io/managed-by=Helm
Annotations: meta.helm.sh/release-name: myapp
meta.helm.sh/release-namespace: myapp
CreationTimestamp: Mon, 26 Apr 2021 07:02:32 +0000
Reference: Deployment/myapp
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 5% (17m) / 75%
Min replicas: 5
Max replicas: 12
Deployment pods: 1 current / 1 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
ading the events when a new rollout placed.
as per the events captured from the “myapp” Namespace , there were new deployment rolled out for myapp as below.
During the new pods creation its showing more CPU spikes as we are getting alert from the Azuremonitor that its exceeds the threshold of 80%.[the “Maximum limit” of 1 core specified in the deployment spec]
30m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-9fmrk
30m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-hfr8w
29m Normal SuccessfulDelete replicaset/myapp-1a2b3c4d5e Deleted pod: myapp-1a2b3c4d5e-l2pnd
31m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 1
30m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 2
30m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 2
30m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 1
30m Normal ScalingReplicaSet deployment/myapp Scaled up replica set myapp-5ddc98fb69 to 3
29m Normal ScalingReplicaSet deployment/myapp Scaled down replica set myapp-1a2b3c4d5e to 0
Alert settings
Period Over the last 15 mins
Value 100.274747
Operator GreaterThan
Threshold 80