I have a service (let's call it video) that uses a lot of UDP ports (each instance uses 400-500 UDP ports). In a normal production environment I would have 4-10 instances, so ~5000 UDP ports maximum. The service is deployed inside EKS as a StatefullSet (each instance listens to different ports based on it's position in the set, pod 1 6001-6500, pod 2 6501-7000 and so on). Inside the EKS cluster I have ~40 other services only 4 of them have ingresses defined.
My original plan was to use Amazon Network Load Balancer for this service, but I've hit a lot of limitations that make this impossible. This is my current service definition.
apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: external
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: eipalloc-yyyyyyyyyyyyy
creationTimestamp: null
labels:
io.kompose.service: videoserver-udp
name: videoserver-udp
spec:
selector:
io.kompose.service: videoserver
type: LoadBalancer
ports:
- name: "3478"
port: 3478
protocol: UDP
targetPort: 3478
- name: "6001"
port: 6001
protocol: UDP
targetPort: 6001
- name: "6002"
port: 6002
protocol: UDP
targetPort: 6002
Main NLB limitation is the I can't define more than 60 rules (500 maximum with increased quote) and that I can't define both UDP and TCP for the same port (3478 would be a STUN/TURN port that uses both TCP and UDP).
I can't find any documentation for this, but ideally I would like to create my own "proxy" for this service. One or more pods (or EC2 instances inside the same subnet if that's possible) that based on the ports send the traffic to pod-X from the StatefullSet. These pods should have their own public IP so I wouldn't have to go through the NLB.
Has anyone had a similar problem, or knows AWS well enough to lead me in the right direction? Thanks in advance.