Reactor Flux timeout with RabbitMq MessageListenerContainer

Viewed 792

Using a hot publisher model, the following timeout exception is occurring roughly 50% of the time in a cloud environment:

[ERROR] reactor.core.scheduler.Schedulers                  - Scheduler worker in group main failed with an uncaught exception
reactor.core.Exceptions$ErrorCallbackNotImplemented: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 300000ms (and no fallback has been configured)
Caused by: java.util.concurrent.TimeoutException: Did not observe any item or terminal signal within 300000ms (and no fallback has been configured)
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.handleTimeout(FluxTimeout.java:289)
reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.doTimeout(FluxTimeout.java:274)
reactor.core.publisher.FluxTimeout$TimeoutTimeoutSubscriber.onNext(FluxTimeout.java:396)
reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
datadog.trace.instrumentation.reactor.core.TracingSubscriber.onNext(TracingSubscriber.java:75)
reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:73)
datadog.trace.instrumentation.reactor.core.TracingSubscriber.onNext(TracingSubscriber.java:75)
reactor.core.publisher.MonoDelay$MonoDelayRunnable.run(MonoDelay.java:117)
reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)

However I cannot replicate this locally. My leading theory is the the cause is occurring during the creation of the flux and something sour is happening between Flux and MessageListenerContainer. The creation looks like:

@Bean
public Flux<RabbitEventPublishEnvelope> masterFlux(
        Queue eventQueue,
        ObjectMapper objectMapper,
        MessageListenerContainerFactory messageListenerContainerFactory) {
    log.info("Create a listener for the topic queue: '{}'", eventQueue.getName());
    MessageListenerContainer mlc = messageListenerContainerFactory
            .createDirectMessageListenerContainer(eventQueue.getName());

    log.info("Define the master Flux for event subscriptions on queue '{}'", eventQueue.getName());
    Flux<RabbitEventPublishEnvelope> masterFlux = Flux.create(emitter -> {
        mlc.setupMessageListener(m -> {
            RabbitEventPublishEnvelope payload = null;
            try {
                log.info("Creating payload");
                payload = objectMapper.readValue(m.getBody(), RabbitEventPublishEnvelope.class);
            } catch (IOException e) {
                log.error("Failed to parse RabbitEventPublishEnvelope:\n{}", m.getBody());
                throw new RuntimeException("Failed to parse RabbitEventPublishEnvelope", e);
            } catch (Exception e) {
                log.error("Unhandled exception in Flux.create(): {}", e.getMessage(), e);
            }
            log.info("Emitting payload");
            emitter.next(payload);
        });
        emitter.onRequest(v -> {
            log.info("MLC starting");
            mlc.start();
            log.info("Start recipe event subscription");
        });
        emitter.onDispose(() -> {
            // WARNING: DO NOT issue `mlc.stop();` here or it will cause responses to hang.
            // The main reason this callback handler is implemented is to document what will break our implementation.
            log.info("Done with recipe event subscription");
        });
    });

    log.info("Created master flux for queue = '{}'", eventQueue.getName());
    return masterFlux
            .log("Publishing flux")
            .publish()
            .autoConnect()
            .timeout(Duration.ofMillis(10000))
            .doOnError(error -> log.error("Unhandled exception in masterFlux(): {}", error.getMessage(), error))
            .log("Auto connection successful");
}

This exception feels like a configuration mistake, but I cannot narrow down what would be causing this timeout to confirm that. Any help would be appreciated!

2 Answers

Referring to: https://www.codota.com/code/java/methods/reactor.core.publisher.Flux/timeout I suspect the problem is that no timeout has been specified as part of the creation of masterFlux.

Referring to: https://projectreactor.io/docs/core/release/api/reactor/core/publisher/Flux.html

There is a static method for create() with this definition: create(Consumer<? super FluxSink> emitter, FluxSink.OverflowStrategy backpressure)

So, I'm guessing, seeing that the OverflowStrategy has not been defined, if the transportation of the message fails, the code does not know what to do and throws this exception. More about the OverflowStrategy can be found here: https://projectreactor.io/docs/core/release/api/reactor/core/publisher/FluxSink.OverflowStrategy.html

If the behavior differs between hot and cold Flux, perhaps there is something within the Flux default configurations--likely an overridable method during your create() call--for how to identify the error.

If the behavior is the same for both a hot and cold Flux, I would look into the

Another clarification would be whether there are any architecture components in play (e.g., are the subscriber/publisher directly connected, or do they go through various load balancers & proxies), as it could help isolate the error.

Related