I'm trying to load messages from relatively large topic (billion+ records, more then 100 GiB, single partition) using Apache NiFi (nifi-1.11.4-RC1, OpenJDK 8, RHEL7), but performance seems to be far too low:
1248429 messages (276.2 MB) per 5 minutes for ConsumeKafka_2_0 and 295 batches (282.5 MB) for ConsumeKafkaRecord_2_0. I.e. only 4161 messages (920 KB) per second.
Results of kafka-consumer-perf-test.sh (same node, same consumer group and same topic) are more impressive: 263.4 MB (1190937 records) per second. Too much difference for any reasonable overhead.
I've configured cluster according to Best practices for setting up a high performance NiFi installation, but throughput didn't increase.
Each node has 256 GB RAM and 20 cores, Maximum Timer Driven Thread Count is set to 120, but NiFi GUI shows only 1 or 2 active threads, and CPU load is almost zero, so is disk queue.
I've tested several flows, but even ConsumeKafka_2_0 with autoterminated 'success' relationship shows the same speed.
Is it possible to increase performance of these processors? It looks like some artificial limit or throttle, because I couldn't find any bottleneck...
Help, please, I'm completely stuck!
UPD1:
# JVM memory settings
java.arg.2=-Xms10240m
java.arg.3=-Xmx10240m
Scheduling Strategy : Timer driven
Concurrent Tasks : 64
Run Schedule : 0 sec
Execution : All nodes
Maximum Timer Driven Thread Count : 120
Maximum Event Driven Thread Count : 20
UPD2:
When I consume topic with many partitions or several topics together with one ConsumeKafka_2_0 processor, or when I use several processors with different consumer groups with same topic, total throughput increases accordingly.
So, Maximum Timer Driven Thread Count and Concurrent Tasks aren't primary culprits. Problem is somewhere in task scheduling, or in processor itself.