I have set up a new VPC in my AWS account, using their new wizard, which creates all the necessary resources (igw, natgw, public/private subnets, routing tables, etc.)
After spinning up tens of ElasticBeanstalk environments, I have come to conclusion that the instances are not able to reach elasticbeanstalk-health.us-east-1.amazonaws.com, and therefore EB fails the initial deployment of the environment:
The EB configuration:
- Instances are deployed to the public subnets, allocated public IP (they can reach the internet)
- Instances are assigned the default SGs created by EB, and also the VPC's default SG.
- The application load balancer is publicly available, deployed on the public subnets and can reach the instances.
- However, the healthcheck fails because nothing responds from the instance's side, because the EB default app is not yet installed.
- I can verify the load balancer can reach the instance thanks to opening a socket on the relevant port (
nc -l -p 80) and seeing the LB healthcheck requests being received.
Troubleshooting further:
I have SSH'ed into one of the instances and tried connecting EB's health endpoint -- without any success.
Testing from my own laptop, the connection succeeds.
The issue roots to the DNS resolution of the health endpoint:
- From my own computer:
PING elasticbeanstalk-health.us-east-1.amazonaws.com (35.170.164.93) - From the EB instance:
PING elasticbeanstalk-health.us-east-1.amazonaws.com (10.0.154.188) - The difference in IPs is not what bothers me, but what does is that on the instance the resolved IP begins with
10.0.xxx.xxx-- which is configured to be routed to _local_ in the routing tables, as the VPC's default CIDR is10.0.0.0/16.
- From my own computer:
Reminder that this is public subnet, so there's no point in associating VPC Endpoint to EB/EB-health endpoints.
I'm wondering why the EB-health hostname is resolved to such IP address, and by whom, as it definitely should be resolved a public IP address available to the internet.