ECS service desire count get resets by Auto Scaling

Question

I want to stop my ECS Fargate test environment everyday, the best option to me is just to modify the ECS Service to have a desire count to zero as follows aws ecs update-service --cluster COOL --service blablabla --desired-count 0, so any task running will be drained within minutes, it works just fine if I have not an Auto Scaling.

The issue start when that ECS Service has an Auto Scaling, as you can see in the picture below, I have min 1, desired 1 and max 3, if I use the AWS CLI command above, the service reset to 1 after a minute or so, and no tasks will be stopped because the Auto Scaling has a desired 1.

If I manually edit the ECS Service and I set desired 0 for both Service and Auto Scaling (and of course min 0 in Auto Scaling as well) , I can stop any running tasks.

I tried to use the command below for Auto Scaling hoping that I can change the desired but it is not available.

aws application-autoscaling register-scalable-target \
    --service-namespace ecs \
    --scalable-dimension ecs:service:DesiredCount \
    --resource-id service/COOL/blablabla \
    --min-capacity 1 \
    --max-capacity 3

I have been thinking that maybe a way to do this, is to delete the Auto Scaling via AWS CLI then changing the ECS Service desired to 0, so everything will be stop, then the day after to recreate the Auto Scaling and to update the ECS Service desired to 1.

Summarizing, I want to stop/start my ECS Fargate test environment every day, so far editing the ECS Service suits to me, but the Auto Scaling interrupt this action.

Can anyone propose a solution for this?, thanks.

user_14579259 · Answer

If you're using ECS with application autoscaling, you don't need to change number of task related to the service and desired number of tasks related to the autoscaling options. Both this values will be replaced as soon as your autoscaling policy is triggered. Instead, you have to adjust policy itself (python SDK):

    response = app_autoscaling.register_scalable_target(
        ServiceNamespace='ecs',
        ResourceId=f"service/{ECS_CLUSTER_NAME}/{ECS_SERVICE_NAME}",
        ScalableDimension='ecs:service:DesiredCount',
        MinCapacity=0,
        MaxCapacity=0,
        SuspendedState={
            'DynamicScalingInSuspended': False,
            'DynamicScalingOutSuspended': True,
            'ScheduledScalingSuspended': True
        }
    )

With zeros in Capacity you tell App Autoscaling to scale-in capacity down to zero. With DynamicScalingOutSuspended set to True you disallow new scale-out events triggered by TargetTracking policy.

To revert back changes just replace zeros with desired capacity and set suspended options to false (to allow further scaling).

user_13070 · Answer

I would call this "working as expected". You can't manually set the task count like this if you have auto-scaling enabled. If you want to shutdown your service every day, you have two options:

Don't use auto-scaling for the ECS service, and continue setting the desired-count directly.
Instead of updating the desired-count directly, in conflict with the auto-scaling group, update the auto-scaling configuration and let it handle scaling your service down to 0 tasks.

ECS service desire count get resets by Auto Scaling

2 Answers