Java DNS doesn't respect AWS CloudMap round-robin

user_15793470 0
Viewed 305

We are developing Java services with ECS Fargate. We want to use autoscaling so our services are registered in AWS CloudMap for DNS discovery.

I hit a very strange issue today where all the calls were going to the same instance of the service. So I started an EC2 in the same VPC to dig the service :

[ec2-user@.... ~]$ dig +short something.local
10.253.0.199
10.253.1.129
10.253.0.6
[ec2-user@.... ~]$ dig +short something.local
10.253.1.129
10.253.0.6
10.253.0.199
[ec2-user@.... ~]$ dig +short something.local
10.253.0.6
10.253.0.199
10.253.1.129

It shows that the round robin is correctly working on the AWS side...

We disabled the DNS caching and validated the effects with sun.net.InetAddressCachePolicy.get() returning 0.

We read the underlying code in the OpenJDK code, showing that getByName uses the first entry returned by getAllByName

Then I wrote a code that executes java.net.InetAddress.getAllByName("something.local") every second and prints the result :

10.253.1.129
10.253.0.6
10.253.0.199
========
10.253.1.129
10.253.0.199
10.253.0.6
========
10.253.1.129
10.253.0.6
10.253.0.199
========

We can see here that the result returns always the same address in the first place but the 2 other one are well-balanced...

I tried to find the logic in the source code of the InetAddress class in the openjdk-11 repo (https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.base/share/classes/java/net/InetAddress.java) but I didn't see anything going wrong...

Do you have any solution about that ?

amazon-web-services dns java
2 Answers

If changing the cache policy is not working properly, you can change the TTL for DNS cache entries setting the networkaddress.cache.ttl property with:

// Cache elements for one second
java.security.Security.setProperty("networkaddress.cache.ttl" , "1")
user_9734180

We have encountered the same problem. We also have configured the jvm security property for the networkaddress.cache.ttl=0 but it appears the dns cache configuration is not respected. We also looked at the OS level caching but we can not seem to find the dns caching service used by openjdk:11-jre-slim-buster as our base image.

Our use case, might not be the same with yours but hoping some insights might help. What we did is to have some workaround to be able to still use AWS Service Discovery by using dnsjava.

We created a test service and we tried getting the address using

java.net.InetAddress.getByName("test-service.dev");

The result does not return different IPs as we suppose to expect.

1 TTL: 0, ADDRESS: xx.x.97.26

2 TTL: 0, ADDRESS: xx.x.97.26

3 TTL: 0, ADDRESS: xx.x.97.26

4 TTL: 0, ADDRESS: xx.x.97.26

5 TTL: 0, ADDRESS: xx.x.97.26

What we did is to use dnsjava and get the ip address returned by dnsjava's

 Address.getByName("test-service.dev");

We had the following result:

1 TTL: 0, ADDRESS: xx.x.97.26

2 TTL: 0, ADDRESS: xx.x.103.245

3 TTL: 0, ADDRESS: xx.x.97.26

4 TTL: 0, ADDRESS: xx.x.103.245

5 TTL: 0, ADDRESS: xx.x.97.26

6 TTL: 0, ADDRESS: xx.x.103.245

The result shows that different IP addresses are returned. We then route our request to the ip address returned.

user_188580820
Related