I'm trying to learn autoscaling for ECS with EC2 launch type.
Without the autoscaling part, everything works well.
When I add the autoscaling part, or the Scalable Target the Alarm and Policy for both, scaling in and out, the service gets stuck in the event:
service ecs-service was unable to place a task because no container instance met all of its requirements. The closest matching container-instance XXX has insufficient CPU units available.
If I look at the service the desired capacity is stuck in 4, pending is 0 and running is 1.
In relation to the alarms, the high cpu usage alarm is OK and the low cpu usage alarm is In alarm.
The Task Definition has 1024 MB assigned to CPU and 1024 MB to Memory.
The Container has 1024 MB assigned to CPU and 1024 MB to Memory.
And I have been waiting for more than 40 minutes.
What would I expect?
I'm setting a low threshold for high CPU Usage (20%) to make the alarm react easily.
Then, I increase the quantity up to 4, checking the used CPU percentage.
This should work in both ways, when is adding and when it is removing. So, it should add up to 4 when high is enabled and go down to 1 when low is enabled.
Here's the entire chain of events without tasks ids, dates and events ids to simplify its reading.
service ecs-service was unable to place a task because no container instance met all of its requirements. The closest matching container-instance XXX has insufficient CPU units available. For more
service ecs-service registered 1 targets in target-group ecs-target
service ecs-service was unable to place a task because no container instance met all of its requirements. The closest matching container-instance XXX has insufficient CPU units available. For more information, see the Troubleshooting section.
Message: Successfully set desired count to 4. Waiting for change to be fulfilled by ecs. Cause: monitor alarm high-cpu-usage in state ALARM triggered policy ecs-high-policy
service ecs-service has started 1 tasks: task
service ecs-service has stopped 1 running tasks: task
service ecs-service deregistered 1 targets in target-group ecs-target
service ecs-service (instance XXX) (port 8080) is unhealthy in target-group ecs-target due to (reason Health checks failed)
service ecs-service has started 1 tasks: task
service ecs-service was unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster. For more information, see the Troubleshooting section.
Message: Successfully set desired count to 4. Found it was later changed to 0. Cause: monitor alarm high-cpu-usage in state ALARM triggered policy ecs-high-policy
Message: Successfully set desired count to 4. Found it was later changed to 0. Cause: monitor alarm high-cpu-usage in state ALARM triggered policy ecs-high-policy
Message: Successfully set desired count to 3. Change successfully fulfilled by ecs. Cause: monitor alarm high-cpu-usage in state ALARM triggered policy ecs-high-policy
Message: Successfully set desired count to 2. Change successfully fulfilled by ecs. Cause: monitor alarm high-cpu-usage in state ALARM triggered policy ecs-high-policy
This is my Scalable Target, Alarms and Policies:
The service uses a Load Balancer.
ServiceScalableTarget:
Type: AWS::ApplicationAutoScaling::ScalableTarget
DependsOn: Service
Properties:
MaxCapacity: !Ref MaxSize
MinCapacity: !Ref MinSize
ResourceId:
Fn::Join:
- '/'
- - 'service'
- Ref: Cluster
- Fn::GetAtt:
- Service
- 'Name'
RoleARN:
Fn::ImportValue: !Ref ECSAutoScalingRole
ScalableDimension: ecs:service:DesiredCount
ServiceNamespace: ecs
HighCpuUsageAlarm:
Type: AWS::CloudWatch::Alarm
DependsOn: ScalingPolicyHigh
Properties:
AlarmName: high-cpu
MetricName: CPUUtilization
Namespace: AWS/ECS
Dimensions:
- Name: ServiceName
Value: !Ref ServiceName
- Name: ClusterName
Value: !Ref Cluster
Statistic: Average
Period: 300
EvaluationPeriods: 1
Threshold: 20
ComparisonOperator: GreaterThanOrEqualToThreshold
AlarmActions:
- !Ref ScalingPolicyHigh
ScalingPolicyHigh:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: olicy-high
PolicyType: StepScaling
ScalingTargetId:
Ref: ServiceScalableTarget
StepScalingPolicyConfiguration:
AdjustmentType: ChangeInCapacity
Cooldown: 600
MetricAggregationType: Average
StepAdjustments:
- MetricIntervalLowerBound: 0
MetricIntervalUpperBound: 15
ScalingAdjustment: 1
- MetricIntervalLowerBound: 15
MetricIntervalUpperBound: 25
ScalingAdjustment: 2
- MetricIntervalLowerBound: 25
ScalingAdjustment: 3
LowCpuUsageAlarm:
Type: AWS::CloudWatch::Alarm
DependsOn: ScalingPolicyLow
Properties:
AlarmName: low-cpu
MetricName: CPUUtilization
Namespace: AWS/ECS
Dimensions:
- Name: ServiceName
Value: !Ref ServiceName
- Name: ClusterName
Value: !Cluster
Statistic: Average
Period: 300
EvaluationPeriods: 2
Threshold: 15
ComparisonOperator: LessThanOrEqualToThreshold
AlarmActions:
- !Ref ScalingPolicyLow
ScalingPolicyLow:
Type: AWS::ApplicationAutoScaling::ScalingPolicy
Properties:
PolicyName: policy-low
PolicyType: StepScaling
ScalingTargetId:
Ref: ServiceScalableTarget
StepScalingPolicyConfiguration:
AdjustmentType: ChangeInCapacity
Cooldown: 600
MetricAggregationType: Average
StepAdjustments:
- MetricIntervalLowerBound: -15
MetricIntervalUpperBound: 0
ScalingAdjustment: -1
- MetricIntervalLowerBound: -25
MetricIntervalUpperBound: -15
ScalingAdjustment: -2
- MetricIntervalUpperBound: -25
ScalingAdjustment: -3
I'd appreciate help. I cannot make it work properly.