I would start by looking to see if the problem is something else than a GC tuning issue. Here are a couple of possibilities:
A hard memory leak will cause GC times to increase. The work done by a GC is dominated by tracing and copying of reachable objects. If you have a leak, then more and more objects will be (incorrectly) reachable.
A cache that that is keeping too many objects reachable will also increase GC times.
Excessive use of Reference types, finalizers, etc may increase GC times.
I would enable GC logging, and look for patterns in memory and space utilization reported by the GC. If you suspect a memory leak because memory utilization is trending higher in the long term, go to the next step and use a memory profile to track down the leak.
Either way, it is important to understand what is causing the problem before trying to fix it.
After some further analysis, it turns out there was indeed some increase in the amount of data per sec. One of the topics had increased message input from around 10 KBps to 200 KBps. But I believed that Kafka could easily handle this much of data.
It most likely can. However, a 20x increase in throughput will inevitably lead to more objects being created and discarded ... and the GC will need to run more often to deal with this.
How come just 200 Kbps of data divided among 5 brokers was able to break GC.
What makes you think that you have "broken" the GC? 15% time in GC doesn't mean it is broken.
Now, I can imagine that the GC may have difficulty meeting your 20ms max pause time goal, and may be triggering occasional full GCs as a result. Your pause time goal is "ambitious", especially if the heap may grow to 10GB. I would suggest reducing the heap size, increasing the pause time goal, and/or increasing the number of physical cores available to the JVM(s).
By breaking I mean an increased delay in committing offsets and other producer and consumer offsets.
So ... you are just concerned that a 20 x increase in load has resulted in the GC using up to 15% of available CPU. Well that's NOT broken. That is (IMO) expected. The garbage collector is not magic. It needs to use CPU time to do its work. The more work it has to do, the more CPU it needs to use to do it. If your application's workload involves a lot of object allocation, then the GC has to deal with that.
In addition to the tuning ideas above, I suspect that you should set the G1HeapRegionSize size a lot smaller. According to "Garbage First Garbage Collector Tuning" by Monica Beckwith, the default is to have 2048 regions based on the minimum heap size. But your setting will give 1G / 16M == 64 initial regions.
Finally, if your overall goal is to reduce the CPU utilization of the GC, then you should be using the Throughput GC, not G1GC. This will minimize GC overheads. The downside is that GC pause minimization is no longer a goal, so occasional lengthy pauses are to be expected.
And if you plan to stay with G1GC, it is advisable to use the latest version of Java; i.e. Java 11. (See "G1 Garbage Collector is mature in Java 9, finally")
