keycloak cluster on AWS (ECS) - tasks can't communicate

user_12862933 0
Viewed 2078

I'm trying to set up a keycloak cluster in AWS ECS.

I tried JDBC_PING, DNS_PING, ... but I get nothing to work. This was my DNS_PING docker file:

FROM jboss/keycloak:11.0.0
EXPOSE 8080
EXPOSE 7600
EXPOSE 7800
EXPOSE 9000

COPY themes /opt/jboss/keycloak/themes

#Database
ENV DB_VENDOR=mysql
ENV DB_DATABASE=....
ENV DB_ADDR=....
ENV DB_PORT=3306
ENV JDBC_PARAMS='useSSL=false'
ENV PROXY_ADDRESS_FORWARDING=true
ENV JGROUPS_DISCOVERY_PROTOCOL=dns.DNS_PING
ENV JGROUPS_DISCOVERY_PROPERTIES='dns_query=my.services.internal,dns_record_type=A'
ENV JGROUPS_TRANSPORT_STACK=tcp


#Admin user
ENV KEYCLOAK_USER=...
ENV KEYCLOAK_PASSWORD=...

It seems to pick it up, but the two ECS tasks can't communicate with each other: org.jgroups.protocols.pbcast.gms timed out

I have these entries in my security group: enter image description here

So the tcp/udp ports should not be blocked. (They are even opened up within the task definition).

Any idea on how to get this fixed?

amazon-ecs keycloak docker amazon-ec2
1 Answers

First of all: nothing of the pings did work (I tried TCP_PING, JDBC_PING, DNS_PING, S3_PING, ...)

Solution:

What I noticed was that the server log on start up was stating that it was going to start in cluster mode, but that it was not able to communicate with the other server. As this then seemed TCP/UDP communication related within AWS, I started reading more over there network interfaces.

I came up with the network interface of the task definition: by default in ECS it is docker (bridge). This means that the ip addresses are only known per EC2 instance running docker. (I have 2 EC2 instances running in my ECS cluster and each keycloak instance was placed on one). I found out that there is something like awsvpc that you can use (e.g. for service discovery) as a network interface for your task definition.

This awsvpc will spin up a local DNS domain, over all EC2 instances within that ECS cluster. When I did that, the two keycloak instances started to communicate with each other and the issue was solved.

Short answer:

Use awsvpc as network interface for you task definition

user_128629330
Related