How can I know who call System.gc() in spark streaming program?

user_10569558 0
Viewed 306

The GC time is too long in my spark streaming programme. In the GC log, I found that Someone called System.gc() in the programme. I do not call System.gc() in my code. So the caller should be the api I used.

I add -XX:-DisableExplicitGC to JVM and fix this problem. However, I want to know who call the System.gc().

I tried some methods.

  1. Use jstack. But the GC is not so frequent, it is difficult to dump the thread that call the method.
  2. I add trigger that add thread dump when invoke method java.lang.System.gc() in JProfiler. But it doesn't seem to work.

How can I know who call System.gc() in spark streaming program?

spark-streaming jvm garbage-collection
1 Answers

You will not catch System.gc with jstack, because during stop-the-world pauses JVM does not accept connections from Dynamic Attach tools, including jstack, jmap, jcmd and similar.

It's possible to trace System.gc callers with async-profiler:

  1. Start profiling beforehand:

    $ profiler.sh start -e java.lang.System.gc <pid>

  2. After one or more System.gc happens, stop profiling and print the stack traces:

    $ profiler.sh stop -o traces <pid>

    Example output:

    --- Execution profile ---
Total samples       : 6

Frame buffer usage  : 0.0007%

--- 4 calls (66.67%), 4 samples
  [ 0] java.lang.System.gc
  [ 1] java.nio.Bits.reserveMemory
  [ 2] java.nio.DirectByteBuffer.<init>
  [ 3] java.nio.ByteBuffer.allocateDirect
  [ 4] Allocate.main

--- 2 calls (33.33%), 2 samples
  [ 0] java.lang.System.gc
  [ 1] sun.misc.GC$Daemon.run

    In the above example, System.gc is called 6 times from two places. Both are typical situations when JDK internally forces Garbage Collection.

    The first one is from java.nio.Bits.reserveMemory. When there is not enough free memory to allocate a new direct ByteBuffer (because of -XX:MaxDirectMemorySize limit), JDK forces full GC to reclaim unreachable direct ByteBuffers.

    The second one is from GC Daemon thread. This is called periodically by Java RMI runtime. For example, if you use JMX remote, periodic GC is automatically enabled once per hour. This can be tuned with -Dsun.rmi.dgc.client.gcInterval system property.

user_34484190
Related