What is 'capacity' parameter when talking about Flink async IO?

user_1021423 0
Viewed 572

When using Flink AsyncDataStream#unorderedWait, there's a parameter called 'capacity', quote from flink official doc,

Capacity: This parameter defines how many asynchronous requests may be in progress at the same time. Even though the async I/O approach leads typically to much better throughput, the operator can still be the bottleneck in the streaming application. Limiting the number of concurrent requests ensures that the operator will not accumulate an ever-growing backlog of pending requests, but that it will trigger backpressure once the capacity is exhausted.

I'm not quite get it, is it for the whole job, or it's for a subtask?

Let's say my toy flink app consumes a kafka, and for each kafka message, it makes a http request, when it receives the http response, it sinks it to another kafka topic.

And in this example, the parallelism of kafka source to 50, if I set the 'capacity' to 10, what does that mean? Does it mean that the whole app will make at most 10 http requests at the same time? Or, 10 http requests for each subtask (that results in at most 500 http requests at the same time)?

And another question is, what it the best practice of set the 'capacity' in this scenario?

Many thanks!

apache-flink
1 Answers

The capacity is per instance of the async i/o operator. So in your example, there would be at most 500 concurrent http requests.

You may have to do some benchmarking experiments to see where it makes sense to balance the tradeoffs for your use case. If the capacity is too small then under load you're likely to create backpressure prematurely; if capacity is too large, then under load you're likely to overwhelm the external service, leading to timeouts or other errors.

user_20008230
Related