FeignRetryableException When Service is down

I am using OpenFeign client in Spring Boot without using Ribbon or Eureka. I created a custom error decoder which handles response errors as intended but connection refused errors seem to bypass my custom decoder.

P.S. When my remote service is up, I can make requests and receive responses.

I am new to Java and Spring and I am wondering if I need to wrap all my calls with try catch, or adding my custom error handler should be catching the error since it seems cleaner to handle all errors in one place

public class FeignErrorDecoder implements ErrorDecoder {
    private final ErrorDecoder defaultErrorDecoder = new Default();

    @Override
    public Exception decode(String methodKey, Response response) {

        if (response.status() >= 400 && response.status() <= 499) {
            //handle with custom exception
        }

        if (response.status() >=500) {
            //handle with custom exception
        }

       return defaultErrorDecoder.decode(methodKey, response);
    }
}

@Configuration
public class FeignConfig {

  //other beans here
    @Bean
    public ErrorDecoder feignErrorDecoder() {
        return new FeignErrorDecoder();
    }
}