AWS ECS Service restarting bevause of failed Target Group healthcheck

user_11377807 0
Viewed 1786

I have a .net core API which runs inside a Docker container. This container got deployed to the Amazon ECR where I run it with a Task definition (works already)

  • snipet from my Task definition
 "portMappings": [
    {
      "hostPort": 50598,
      "protocol": "tcp",
      "containerPort": 50598
    }
  ],

When I start the service, the task runs and it works fine. I get my public IP where I can check if my calls are available: enter image description here

  • HTTP 401 is ok because the call checks for a valid token

enter image description here

enter image description here

The problem is when I try to add a Load balancer with a target group.

For this I delete my old service and create a new one with a Load Balancer and Target group enter image description here

After I start my service like this the Target Group health check response with a "Request Timeout" and keeps restarting my service. When I check for the public IP of the API it still works, only when I try to access my API through the loadbalancer it doesn't work.

Target Group: enter image description here

Lb: enter image description here enter image description here

aws-application-load-balancer aws-fargate amazon-ecs amazon-web-services
3 Answers

You're getting request timeout likely because the security group attached to the task is not allowing inbound access from the load balancer nodes.

By default even if all nodes fail health checks the load balancer will attempt to forward to all nodes, which with a failed inbound evaluation would cause the timeout.

As long as the host allows port 50598 to the load balancer no timeout should occur, if you're using an ALB you can add a source as the security group attached to the load balancer.

Once this access is working, you will need to ensure your health checks are successful for the HTTP status code and path.

It is also worth noting for an Application Load Balancer you can use ranges from 200 - 499, whereas for a Network Load Balancer it can only have health checks from 200 - 399.

Your target group must be configured to run on port 50598 too, it is currently targeted port 80.

Update

The application appeared to be running on port 80. Configuration for the target group and task was updated to use the port. This then began working again.

user_134609330

Just add the 401 in the Healthcheck Success codes and it should work. As you can define multiple HTTP status codes in the health check of the target group.

enter image description here

Also to deal with "Request Timeout", allow ECS instance Port 50598 from LB

user_32888900

Looking at the logs for my service I found the following reason for stopping it

service my-service (port 80) is unhealthy in target-group my-target-group due to (reason Health checks failed with these codes: [302]).

And Http code 302 is for performing URL redirection.

Since there is nothing wrong with the service itself and it is the failed health checkup that the task is stopped and restarted. I changed the health checkup configuration to consider 200(default) and 302 response code as success. And it is fixed now.

But not sure if this is the right fix for this.

user_40570380
Related