Why is ping timing out from WebServer01 to DBServer?

user_13745430 0
Viewed 317

I have created a Custom VPC using the Terraform code below.

I have applied the Terraform module below, which:

  1. Adds Public and Private Subnets
  2. Configures IGW
  3. Adds NAT Gateway
  4. Adds SGs for both Web and DB Servers

After applying this, I am not able to ping/ssh from public ec2 instance to the private ec2 instance.

Not sure what is missing.

# Custom VPC
resource "aws_vpc" "MyVPC" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "default" # For Prod use "dedicated"

  tags = {
    Name = "MyVPC"
  }
}

# Creates "Main Route Table", "NACL" & "default Security Group"

# Create Public Subnet, Associate with our VPC, Auto assign Public IP
resource "aws_subnet" "PublicSubNet" {
  vpc_id                  = aws_vpc.MyVPC.id # Our VPC
  availability_zone       = "eu-west-2a"     # AZ within London, 1 Subnet = 1 AZ
  cidr_block              = "10.0.1.0/24"    #  Check using this later > "${cidrsubnet(data.aws_vpc.MyVPC.cidr_block, 4, 1)}"
  map_public_ip_on_launch = "true"           # Auto assign Public IP for Public Subnet
  tags = {
    Name = "PublicSubNet"
  }
}

# Create Private Subnet, Associate with our VPC
resource "aws_subnet" "PrivateSubNet" {
  vpc_id            = aws_vpc.MyVPC.id # Our VPC
  availability_zone = "eu-west-2b"     # AZ within London region, 1 Subnet = 1 AZ
  cidr_block        = "10.0.2.0/24"    #  Check using this later > "${cidrsubnet(data.aws_vpc.MyVPC.cidr_block, 4, 1)}"
  tags = {
    Name = "PrivateSubNet"
  }
}

# Only 1 IGW per VPC
resource "aws_internet_gateway" "MyIGW" {
  vpc_id = aws_vpc.MyVPC.id
  tags = {
    Name = "MyIGW"
  }
}

# New Public route table, so we can keep "default main" route table as Private. Route out to MyIGW
resource "aws_route_table" "MyPublicRouteTable" {
  vpc_id = aws_vpc.MyVPC.id # Our VPC

  route {                    # Route out IPV4
    cidr_block = "0.0.0.0/0" # IPV4 Route Out for all
    # ipv6_cidr_block = "::/0"        The parameter destinationCidrBlock cannot be used with the parameter destinationIpv6CidrBlock # IPV6 Route Out for all
    gateway_id = aws_internet_gateway.MyIGW.id # Target : Internet Gateway created earlier
  }
  route {                                           # Route out IPV6
    ipv6_cidr_block = "::/0"                        # IPV6 Route Out for all
    gateway_id      = aws_internet_gateway.MyIGW.id # Target : Internet Gateway created earlier
  }
  tags = {
    Name = "MyPublicRouteTable"
  }
}

# Associate "PublicSubNet" with the public route table created above, removes it from default main route table
resource "aws_route_table_association" "PublicSubNetnPublicRouteTable" {
  subnet_id      = aws_subnet.PublicSubNet.id
  route_table_id = aws_route_table.MyPublicRouteTable.id
}

# Create new security group "WebDMZ" for WebServer
resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id # Our VPC : SGs cannot span VPC

  ingress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  ingress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # You can use Load Balancer
  }
  ingress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }
  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  egress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }
}

# Create new EC2 instance (WebServer01) in Public Subnet
# Get ami id from Console
resource "aws_instance" "WebServer01" {
  ami           = "ami-01a6e31ac994bbc09"
  instance_type = "t2.micro"
  subnet_id     = aws_subnet.PublicSubNet.id
  key_name = "MyEC2KeyPair"   # To connect using key pair
  security_groups = [aws_security_group.WebDMZ.id]    # Assign WebDMZ security group created above
  # vpc_security_group_ids = [aws_security_group.WebDMZ.id]
  tags = {
    Name = "WebServer01"
  }
}

# Create new security group "MyDBSG" for WebServer
resource "aws_security_group" "MyDBSG" {
  name        = "MyDBSG"
  description = "Allows Public WebServer to Communicate with Private DB Server"
  vpc_id      = aws_vpc.MyVPC.id # Our VPC : SGs cannot span VPC

  ingress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"]    # Public Subnet CIDR block, can be "WebDMZ" security group id too as below
    security_groups = [aws_security_group.WebDMZ.id]        # Tried this as above was not working, but still doesn't work
  }
  ingress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvKey.pem     Private Key pasted in MyPvKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description = "Allows HTTP requests: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description      = "Allows HTTPS requests : IPV4"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  ingress {
    description      = "Allows MySQL/Aurora requests"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"] # Public Subnet CIDR block, can be "WebDMZ" security group id too
  }
  egress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvtKey.pem     Private Key pasted in MyPvKey.pem chmod 400 MyPvtKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description = "Allows HTTP requests: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description      = "Allows HTTPS requests : IPV4"
    from_port        = 443
    to_port          = 443
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
  egress {
    description      = "Allows MySQL/Aurora requests"
    from_port        = 3306
    to_port          = 3306
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }
}


# Create new EC2 instance (DBServer) in Private Subnet, Associate "MyDBSG" Security Group
resource "aws_instance" "DBServer" {
  ami           = "ami-01a6e31ac994bbc09"
  instance_type = "t2.micro"
  subnet_id     = aws_subnet.PrivateSubNet.id
  key_name = "MyEC2KeyPair"   # To connect using key pair
  security_groups = [aws_security_group.MyDBSG.id] # THIS WAS GIVING ERROR WHEN ASSOCIATING
  # vpc_security_group_ids = [aws_security_group.MyDBSG.id]
  tags = {
    Name = "DBServer"
  }
}

# Elastic IP required for NAT Gateway
resource "aws_eip" "nateip" {
  vpc = true
  tags = {
    Name = "NATEIP"
  }
}

# DBServer in private subnet cannot access internet, so add "NAT Gateway" in Public Subnet
# Add Target as NAT Gateway in default main route table. So there is route out to Internet.
# Now you can do yum update on DBServer

resource "aws_nat_gateway" "NATGW" {         # Create NAT Gateway in each AZ so in case of failure it can use other
  allocation_id = aws_eip.nateip.id          # Elastic IP allocation
  subnet_id     = aws_subnet.PublicSubNet.id # Public Subnet

  tags = {
    Name = "NATGW"
  }
}

# Main Route Table add NATGW as Target

resource "aws_default_route_table" "DefaultRouteTable" {
  default_route_table_id = aws_vpc.MyVPC.default_route_table_id

  route {
    cidr_block     = "0.0.0.0/0"              # IPV4 Route Out for all
    nat_gateway_id = aws_nat_gateway.NATGW.id # Target : NAT Gateway created above
  }

  tags = {
    Name = "DefaultRouteTable"
  }
}

Why is ping timing out from WebServer01 to DBServer?

amazon-vpc terraform amazon-web-services
2 Answers

There are no specific NACLs and the default NACLs are wide open, so they should not be relevant here.

For this to work the security group on DBServer needs to allow egress to the security group of DBServer or to a CIDR that includes it.

aws_instance.DBServer uses aws_security_group.MyDBSG.

aws_instance.WebServer01 uses aws_security_group.WebDMZ.

The egress rules on aws_security_group.WebDMZ are as follows:

  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # SSH restricted to my laptop public IP <My PUBLIC IP>/32
  }
  egress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
  egress {
    description      = "Allows HTTP requests for VPC: IPV6"
    from_port        = 80
    to_port          = 80
    protocol         = "tcp"
    ipv6_cidr_blocks = ["::/0"]
  }

Those egress rules mean:

  1. Allow SSH to all IPv4 hosts
  2. Allow HTTP to all IPv4 and IPv6 hosts

ICMP isn't listed, so the ICMP echo request will be dropped before it leaves aws_security_group.WebDMZ. This should be visible as a REJECT in the VPC FlowLog for the ENI of aws_instance.WebServer01.

Adding this egress rule to aws_security_group.WebDMZ should fix that:

  egress {
    description      = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port        = -1
    to_port          = -1
    protocol         = "icmp"
    cidr_blocks      = ["10.0.2.0/24"]
  }

DBServer may NOT respond to ICMP, so you may still see timeouts after making this change. Referencing the VPC FlowLog will help determine the difference. If you see ACCEPTs in the VPC FlowLog for the ICMP flows, then the issue is that DBServer doesn't respond to ICMP.

Nothing in aws_security_group.WebDMZ prevents SSH, so the problem with that must be elsewhere.

The ingress rules on aws_security_group.MyDBSG are as follows.

  ingress {
    description = "Allows ICMP requests: IPV4" # For ping,communication, error reporting etc
    from_port   = -1
    to_port     = -1
    protocol    = "icmp"
    cidr_blocks = ["10.0.1.0/24"]    # Public Subnet CIDR block, can be "WebDMZ" security group id too as below
    security_groups = [aws_security_group.WebDMZ.id]        # Tried this as above was not working, but still doesn't work
  }
  ingress {
    description      = "Allows SSH requests: IPV4" # You can SSH from WebServer01 to DBServer, using DBServer private ip address and copying private key to WebServer
    from_port        = 22                          # ssh ec2-user@Private Ip Address -i MyPvKey.pem     Private Key pasted in MyPvKey.pem
    to_port          = 22                          # Not a good practise to use store private key on WebServer, instead use Bastion Host (Hardened Image, Secure) to connect to Private DB
    protocol         = "tcp"
    cidr_blocks = ["10.0.1.0/24"]
  }

Those egress rules mean:

  1. Allow all ICMP from the Public Subnet (10.0.1.0/24)
  2. Allow SSH from the Public Subnet (10.0.1.0/24)

SSH should be working. DBServer likely does not accept SSH connections.

Assuming your DBServer is 10.0.2.123, then if SSH is not running it would look like this from WebServer01 running ssh -v 10.0.2.123:

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 47: Applying options for *
debug1: Connecting to 10.0.2.123 [10.0.2.123] port 22.
debug1: connect to address 10.0.2.123 port 22: Operation timed out
ssh: connect to host 10.0.2.123 port 22: Operation timed out

Referencing the VPC FlowLog will help determine the difference. If you see ACCEPTs in the VPC FlowLog for the SSH flows, then the issue is that DBServer doesn't accept SSH connections.

Enabling VPC Flow Logs

Since this has changed a few times over the years, I will point you to AWS' own maintained, step by step guide for creating a flow log that publishes to CloudWatch Logs.

I recommend CloudWatch over S3 at the moment since querying using CloudWatch Logs Insights is pretty easy compared to setting up and querying S3 with Athena. You just pick the CloudWatch Log Stream you used for your Flow Log and search for the IPs or ports of interest.

This example CloudWatch Logs Insights query will get the most recent 20 rejects on eni-0123456789abcdef0 (not a real ENI, use the actual ENI ID you are debugging):

fields @timestamp,@message
| sort @timestamp desc
| filter @message like 'eni-0123456789abcdef0'
| filter @message like 'REJECT'
| limit 20

In the VPC Flow Log a missing egress rule shows up as a REJECT on the source ENI.

In the VPC Flow Log a missing ingress rule shows up as a REJECT on the destination ENI.

Security Groups are stateful

Stateless packet filters require you to handle arcane things like the fact that most (not all) OSes use ports 32767-65535 for reply traffic in TCP flows. That's a pain, and it makes NACLs (which a stateless) a huge pain.

A stateful firewall like Security Groups tracks connections (the state in stateful) automatically, so you just need to allow the service port to the destination's SG or IP CIDR block in the source SG's egress rules and from the source SG or IP CIDR block in the destination SG's ingress rules.

Even though Security Groups (SGs) are stateful, they are default deny. This includes the outbound initiated traffic from a source. If a source SG does not allow traffic outbound to a destination, it's not allowed even if the destination has an SG that allows it. This is a common misconception. SG rules are not transitive, they need to be made on both sides.

AWS's Protecting Your Instance with Security Groups video explains this very well and visually.

Aside: Style recommendation

You should use aws_security_group_rule resources instead of inline egress and ingress rules.

NOTE on Security Groups and Security Group Rules: Terraform currently provides both a standalone Security Group Rule resource (a single ingress or egress rule), and a Security Group resource with ingress and egress rules defined in-line. At this time you cannot use a Security Group with in-line rules in conjunction with any Security Group Rule resources. Doing so will cause a conflict of rule settings and will overwrite rules.

Take this old-style aws_security_group with inline rules:

resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id

  ingress {
    description = "Allows HTTP requests for VPC: IPV4"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]   # You can use Load Balancer
  }

  egress {
    description = "Allows SSH requests for VPC: IPV4"
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

And replace it with this modern-style aws_security_group with aws_security_group_rule resources for each rule:

resource "aws_security_group" "WebDMZ" {
  name        = "WebDMZ"
  description = "Allows SSH & HTTP requests"
  vpc_id      = aws_vpc.MyVPC.id
}

resource "aws_security_group_rule" "WebDMZ_HTTP_in" {
  security_group_id = aws_security_group.WebDMZ.id

  type        = "ingress"
  description = "Allows HTTP requests for VPC: IPV4"
  from_port   = 80
  to_port     = 80
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}

resource "aws_security_group_rule" "WebDMZ_SSH_out" {
  security_group_id = aws_security_group.WebDMZ.id

  type        = "egress"
  description = "Allows SSH requests for VPC: IPV4"
  from_port   = 22
  to_port     = 22
  protocol    = "tcp"
  cidr_blocks = ["0.0.0.0/0"]
}
user_1545270

Allowing all outbound rules from "WebDMZ" security group as pasted below resolves the issue.

egress {  # Allow allow traffic outbound, THIS WAS THE REASON YOU WAS NOT ABLE TO PING FROM WebServer to DBServer
    description = "Allows All Traffic Outbound from Web Server" 
    from_port   = 0
    to_port     = 0
    protocol    = -1
    cidr_blocks = ["0.0.0.0/0"] 
  }
user_137454300
Related