I have certain spring boot (V1.5.10) containers running on an AWS ECS cluster consisting of two EC2 instances (ECS Agent V1.33.0 and docker V18.06.1-ce).
These services are load-balanced and for the past week, I am facing this issue where certain containers randomly restart, causing their corresponding tasks to terminate. The containers run for around 1-2 hours and they restart.
I have checked the logs for ECS services, there are no helpful logs, just the ones regarding deregistering and restart of the services.
When I check the tasks in a service, it shows an Essential container in the task stopped error and no other information.
I have checked the docker containers on the EC2 instances using docker ps -a and they show exited Exited (1) 14 minutes ago and so on.
I have checked the logs for the ecs-agent as well, but there is no clear reason regarding the restart in them.
Sometimes, the docker daemon stops working as well and commands like docker ps don't work.
These are some of the logs from the ecs agent
fb1e9358d2cb2f20f481e9ef222503acf5f75090b8 <nil> [] <nil> [{volume d1f29eb945fd5e670e17e370af28cfbd7cac1c35f74dec56dc42fc5f58691d7c /var/lib/docker/volumes/d1f29eb945fd5e670e17e370af28cfbd7cac1c35f74dec56dc42fc5f58691d7c/_data /tmp13 local true }] map[com.amazonaws.ecs.cluster:Test com.amazonaws.ecs.container-name:app1 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-west-2:7879097520:task/Test/2d1b6e48fe6748439fad2549de847af1 com.amazonaws.ecs.task-definition-family:Test-App1 com.amazonaws.ecs.task-definition-version:2] 2020-12-02 13:14:15.155236268 +0000 UTC 2020-12-02 13:14:15.693163539 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } bridge 0xc000b36e00} ContainerStatusChangeEvent}] for container [App1]
2020-12-10T13:10:10Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/2d1b6e48fe6748439fad2549de847af1]: redundant container state change. App1 to RUNNING, but already RUNNING
2020-12-10T13:10:10Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/2d1b6e48fe6748439fad2549de847af1]: task at steady state: RUNNING
2020-12-10T13:10:10Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/2d1b6e48fe6748439fad2549de847af1]: waiting for event for task
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: no longer waiting
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: task at steady state: RUNNING
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: waiting for event for task
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: got container [tracking] event: [RUNNING]
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: handling container change [{RUNNING {52d5eaa86f865c1f7e8777fbb9b3368c96f927e0e215a8dda94703a47457dfa4 <nil> [{8081 57145 0.0.0.0 0}] <nil> [{volume 516048bf9205ee72b2b1f9226fa4827c897e853a7582e751fddf5a246d70f48e /var/lib/docker/volumes/516048bf9205ee72b2b1f9226fa4827c897e853a7582e751fddf5a246d70f48e/_data /tmp2 local true }] map[com.amazonaws.ecs.cluster:Test com.amazonaws.ecs.container-name:tracking com.amazonaws.ecs.task-arn:arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd com.amazonaws.ecs.task-definition-family:Test-Tracking com.amazonaws.ecs.task-definition-version:2] 2020-12-10 12:10:58.810724617 +0000 UTC 2020-12-10 12:10:59.335518545 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } bridge 0xc000b5b100} ContainerStatusChangeEvent}] for container [tracking]
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: redundant container state change. tracking to RUNNING, but already RUNNING
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: task at steady state: RUNNING
2020-12-10T13:10:59Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/1b767a28b694464ea82a65ae21f971cd]: waiting for event for task
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: no longer waiting
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: task at steady state: RUNNING
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: waiting for event for task
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: got container [inventory] event: [RUNNING]
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: handling container change [{RUNNING {5df66bb5e4f563e48aee457310a862edff73f0df67f9c2a329bff0f910261731 <nil> [{10000 57132 0.0.0.0 0}] <nil> [{volume 0684429d8b825c08f26074f37ea0f23026eb500a5b0dca75213b56d22e4442d6 /var/lib/docker/volumes/0684429d8b825c08f26074f37ea0f23026eb500a5b0dca75213b56d22e4442d6/_data /tmp8 local true }] map[com.amazonaws.ecs.cluster:Test com.amazonaws.ecs.container-name:inventory com.amazonaws.ecs.task-arn:arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b com.amazonaws.ecs.task-definition-family:Test-Inventory com.amazonaws.ecs.task-definition-version:2] 2020-12-10 10:21:25.284497581 +0000 UTC 2020-12-10 10:21:26.56942361 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } bridge 0xc000b36700} ContainerStatusChangeEvent}] for container [inventory]
2020-12-10T13:12:37Z [INFO] Managed task [arn:aws:ecs:us-west-2:7879097520:task/Test/e8d3302d694642999668484bc6f99f1b]: redundant container state change. inventory to RUNNING, but already RUNNING
I have a warning regarding the outdated ECS agent in my cluster. Could that be an issue?
Any help is appreciated.