The code below executes all web requests (webClient) in parallel, not respecting the limit I put in parallel(5).
Flux.fromIterable(dataListWithHundredsElements)
.parallel(5).runOn(Schedulers.boundedElastic())
.flatMap(element ->
webClient.post().
.bodyValue(element)
.retrieve()
.bodyToMono(String.class)
.doOnError(err -> element.setError(Utils.toString(err)))
.doOnSuccess(r -> element.setResponse(r))
)
.sequential()
.onErrorContinue((e, v) -> {})
.doOnComplete(() -> updateInDatabase(dataListWithHundresdElements))
.subscribe();
I would like to know if it is possible to execute requests according to the value specified in parallel(5) and how best to do that?
One detail, this code is a Spring MVC application which I am making requests for an external service.
UPDATE 01
In fact Flux creates the 5 threads, however, all requests (WebClient Mono) are executed at the same time.
What I want is to have 5 requests executed at a time, so when 1 request ends another request is started, but at no time should there be more than 5 requests in parallel.
As Mono is also a reactive type, it seems to me that the 5 threads of Flux invoke it and are not blocked, in practice what happens is that all requests happen in parallel.
UPDATE 02 - External Service Logs
This is the log of the external service which takes about 5 seconds to respond. As you can see in the logs below, 14 requests at the same time.
2020-05-08 11:53:56.655 INFO 28223 --- [nio-8080-exec-8] EXTERNAL SERVICE LOG {"id": 21} http-nio-8080-exec-8
2020-05-08 11:53:56.655 INFO 28223 --- [nio-8080-exec-7] EXTERNAL SERVICE LOG {"id": 20} http-nio-8080-exec-7
2020-05-08 11:53:56.659 INFO 28223 --- [nio-8080-exec-2] EXTERNAL SERVICE LOG {"id": 27} http-nio-8080-exec-2
2020-05-08 11:53:56.659 INFO 28223 --- [nio-8080-exec-6] EXTERNAL SERVICE LOG {"id": 19} http-nio-8080-exec-6
2020-05-08 11:53:56.659 INFO 28223 --- [io-8080-exec-10] EXTERNAL SERVICE LOG {"id": 23} http-nio-8080-exec-10
2020-05-08 11:53:56.660 INFO 28223 --- [nio-8080-exec-5] EXTERNAL SERVICE LOG {"id": 18} http-nio-8080-exec-5
2020-05-08 11:53:56.660 INFO 28223 --- [nio-8080-exec-9] EXTERNAL SERVICE LOG {"id": 17} http-nio-8080-exec-9
2020-05-08 11:53:56.660 INFO 28223 --- [nio-8080-exec-1] EXTERNAL SERVICE LOG {"id": 29} http-nio-8080-exec-1
2020-05-08 11:53:56.661 INFO 28223 --- [nio-8080-exec-4] EXTERNAL SERVICE LOG {"id": 24} http-nio-8080-exec-4
2020-05-08 11:53:56.666 INFO 28223 --- [io-8080-exec-11] EXTERNAL SERVICE LOG {"id": 25} http-nio-8080-exec-11
2020-05-08 11:53:56.675 INFO 28223 --- [io-8080-exec-13] EXTERNAL SERVICE LOG {"id": 42} http-nio-8080-exec-13
2020-05-08 11:53:56.678 INFO 28223 --- [io-8080-exec-14] EXTERNAL SERVICE LOG {"id": 28} http-nio-8080-exec-14
2020-05-08 11:53:56.680 INFO 28223 --- [io-8080-exec-12] EXTERNAL SERVICE LOG {"id": 26} http-nio-8080-exec-12
2020-05-08 11:53:56.686 INFO 28223 --- [io-8080-exec-15] EXTERNAL SERVICE LOG {"id": 22} http-nio-8080-exec-15
UPDATE 03 - Reactor Logs
Reinforcing, the external service takes about 5 seconds to respond. However it is possible to see that all requests (14) are made at almost the same time.
2020-05-08 11:53:56.051 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : onSubscribe([Fuseable] FluxPublishOn.PublishOnSubscriber)
2020-05-08 11:53:56.053 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : request(unbounded)
2020-05-08 11:53:56.081 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : onSubscribe([Fuseable] FluxPublishOn.PublishOnSubscriber)
2020-05-08 11:53:56.081 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : request(unbounded)
2020-05-08 11:53:56.082 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : onSubscribe([Fuseable] FluxPublishOn.PublishOnSubscriber)
2020-05-08 11:53:56.082 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : request(unbounded)
2020-05-08 11:53:56.093 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : onSubscribe([Fuseable] FluxPublishOn.PublishOnSubscriber)
2020-05-08 11:53:56.093 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : request(unbounded)
2020-05-08 11:53:56.094 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : onSubscribe([Fuseable] FluxPublishOn.PublishOnSubscriber)
2020-05-08 11:53:56.095 INFO 28223 --- [nio-8080-exec-1] reactor.Parallel.RunOn.1 : request(unbounded)
2020-05-08 11:53:56.110 INFO 28223 --- [oundedElastic-1] reactor.Parallel.RunOn.1 : onNext(@40ddcd53)
2020-05-08 11:53:56.112 INFO 28223 --- [oundedElastic-5] reactor.Parallel.RunOn.1 : onNext(@200e0819)
2020-05-08 11:53:56.112 INFO 28223 --- [oundedElastic-2] reactor.Parallel.RunOn.1 : onNext(@3b81eee2)
2020-05-08 11:53:56.113 INFO 28223 --- [oundedElastic-3] reactor.Parallel.RunOn.1 : onNext(@60af2a4d)
2020-05-08 11:53:56.115 INFO 28223 --- [oundedElastic-4] reactor.Parallel.RunOn.1 : onNext(@723db553)
2020-05-08 11:53:56.440 INFO 28223 --- [oundedElastic-2] reactor.Parallel.RunOn.1 : onNext(@387743b5)
2020-05-08 11:53:56.440 INFO 28223 --- [oundedElastic-3] reactor.Parallel.RunOn.1 : onNext(@62ed2f8d)
2020-05-08 11:53:56.440 INFO 28223 --- [oundedElastic-5] reactor.Parallel.RunOn.1 : onNext(@1a40554a)
2020-05-08 11:53:56.442 INFO 28223 --- [oundedElastic-3] reactor.Parallel.RunOn.1 : onNext(@1bcb696a)
2020-05-08 11:53:56.440 INFO 28223 --- [oundedElastic-4] reactor.Parallel.RunOn.1 : onNext(@46c98823)
2020-05-08 11:53:56.443 INFO 28223 --- [oundedElastic-3] reactor.Parallel.RunOn.1 : onComplete()
2020-05-08 11:53:56.446 INFO 28223 --- [oundedElastic-5] reactor.Parallel.RunOn.1 : onComplete()
2020-05-08 11:53:56.442 INFO 28223 --- [oundedElastic-2] reactor.Parallel.RunOn.1 : onNext(@1c0da4a)
2020-05-08 11:53:56.448 INFO 28223 --- [oundedElastic-2] reactor.Parallel.RunOn.1 : onComplete()
2020-05-08 11:53:56.452 INFO 28223 --- [oundedElastic-4] reactor.Parallel.RunOn.1 : onNext(@14d54d26)
2020-05-08 11:53:56.453 INFO 28223 --- [oundedElastic-4] reactor.Parallel.RunOn.1 : onComplete()
2020-05-08 11:53:56.490 INFO 28223 --- [oundedElastic-1] reactor.Parallel.RunOn.1 : onNext(@46e43af)
2020-05-08 11:53:56.492 INFO 28223 --- [oundedElastic-1] reactor.Parallel.RunOn.1 : onNext(@5ca02355)
2020-05-08 11:53:56.496 INFO 28223 --- [oundedElastic-1] reactor.Parallel.RunOn.1 : onComplete()