Spring Cloud: How to retry with another service if first request fails

I found this guide which explains the new spring-cloud-loadbalancer(netflix stuff is deprecated/in maintanance).

It explains exactly my use case. I need to client load balance between two fixed external servers, which can not be registered to Euraka. So I also can't use service discovery.

The load balancing works fine but I need to retry if the request to the first server fails. The retry should be send to the second server.

But the load balanced RestTemplate just picks the server determined by the load balancing(which uses virtual roundrobin, so it picks sometimes the already used server again).

      @Retryable(
      value = { RestClientException.class},
      maxAttempts = 2,
      backoff = @Backoff(delay = 0))
  public Response callApiEndpoint(Request request) {
        return loadBalancedRestTemplate.postForEntity("https://my-service/path", requestEntity,
            Response.class);
  }

The most tutorials about this refering to netflix ribbon which I don't use. Also I found some hints to implement my own LoadBalancedRetryPolicy.

How can it be achived to do the retry with not the same server again if I use the new spring-cloud-loadbalancer and no service discovery?