Context

Hi everyone, I am working on a side project for practicing Go and distributed systems. A particular one is Twitter's Snowflake implementation - I have written the business logic (here).

Now, I want to deploy this logic across multiple instances to build a highly available API for above. In local, I have written the web server and I have used docker-compose to spawn multiple instances of the server. The server is completely stateless without dependency on any other service or DB.

Problem

The part that I am stuck at is to decide how to expose this cluster of web-servers to the client.

Example - Let's say I have spawned my 5 servers at ip1, ip2, ip3, ip4, ip5.

I have following options that I can think of -

1. Have another web server (let's say ip6) which acts as the load balancer across these 5 servers. Client always calls ip6 and it decides which server and sends request there, get's the response and sends back to client.
   • Pros - Client uses just ip6 for my service.
   • Cons - N/w hop via LB - reduced availability and increased latency. Also LB becomes the bottleneck - what if LB goes down!
2. Have another web server (let's say ip6) which acts as the location provider. Client calls ip6 and in response get's the IPs of available web-servers among the five and client calls that directly to get response. Client caches it and if node becomes down, client calls ip6 again to get new IPs.
   • Pros - Once the IP is received, client communicates with nodes directly. Better availability and latency.
   • Cons - This location provider still is a bottleneck for new clients.
3. A way to register a DNS IP which automatically provides redundancy and can be either linked to >1 instances of LB or directly to nodes.

I am unable to decide among three, IMO 3rd option is unavoidable for true availability (even LB going down). I would highly appreciate if someone can give some inputs on this. Thanks.