After upgrading to Cassandra 3.11.8 we are facing intermittent runaway NTR queue growth.
The NTR queue reaches a size of 5M at which point the node can't recover from the resulting GC overhead.
Having read Improving Apache Cassandra’s Front Door and Backpressure, we tried configuring native_transport_max_concurrent_requests_in_bytes to 1MB, assuming that would prevent the NTR queue from exploding. However, we still saw the NTR queue reach a size above 1M requests (so surely above a size of 1MB).
We have -Dcassandra.max_queued_native_transport_requests=3072 vm argument set, and obviously that doesn't limit the queue size to 3072 requests. I'm also not sure if this setting still applies following this change in 3.11.5.
My questions are:
- What is the right way to limit the NTR queue size in Cassandra 3.11.8?
- If there is such a way, does it apply both to the inbound and outbound NTR queues? Or is the outbound queue really unbounded as stated here?