Elixir: Latency between creation and registry of a GenServer in distributed Horde and calling its API

I'm using Horde 0.8.3 to manage the distributed registry and supervisor of an Elixir application. The application creates UserAgent-s (UA for short) that are implemented as GenServer-s. Each UA represents a user connected to the application. UAs are created with Horde.DynamicSupervisor.start_child(MyApi.DSup, {MyApi.UserAgent, user}). In the same function where the UA is created, the application calls a UA client API, say UserAgent.ping(ua_id) (the nature of this API is not important here). ping(...) uses via_tuple to find the pid of the UA in the registry and sends it a :ping message.

The application is started on two nodes, say N1 and N2. Suppose start_child(...) and ping()... execute on N1 while the UA had been created on N2. When ping(...) is called in these conditions, most of the time it doesn't find the UA in the Horde registry. I did some experiments and found the UA becomes available (in Horde) after about 150ms to 350ms, 250ms on average.

Obviously that time is needed for Horde machinery (deltaCRDT) to have the data dispatched on all nodes of the cluster. That dispatch is done asynchronously in the background.

1. 250ms on average seems a bit long for an app running on the one physical server (no network latency expected) with 12 cores and while the registry is empty. Is this a normal latency or should we dig a bit more to see if there is an issue ?

2. I would have thought the Horde.DynamicSupervisor.start_child(...) was synchronous, that is it returns only once all dispatching was done, as it is supposed to have the same behavior as the standard DynamicSupervisor. Is this assumtion correct and is there a way to have have a synchronous call ?

3. I ended by implementing the function get_pid(id, ...) below that I call to get the pid of a UA. Is this really a good solution ?

   def get_pid(_id, _wait, waited, max_wait) when waited > max_wait, do: :error
   
   def get_pid(id, wait, waited, max_wait) do
     Logger.debug("get_pid(#{id}, wait=#{wait}, waited=#{waited}, max_wait=#{max_wait})")
     case Horde.Registry.lookup(via_tuple(id)) do  [
       {pid, _}] ->
         Logger.debug("\t-> got pid=#{inspect pid}")
         pid
       [] ->
         Logger.debug("\t-> got []. Going for a nap for #{wait}ms")
         Process.sleep(wait)
         get_pid(id, wait*2, waited+wait, max_wait)
     end
   end