We have our application deployed at customer premises. Out of multiple installations, one instance is constantly crashing while executing the native code as mentioned in the title.
The crashes are observed at very uneven intervals ranging from 4 hours to even more than 5 days. All the occurrences show that the crash happens while executing the Java frame: io.netty.internal.tcnative.SSL.readFromSSL and native library and offset of [libc.so.6+0x81a8c]. Java frames look similar in most of the cases.
Dependencies:
netty-all-4.1.50.Final.jar
netty-tcnative-2.0.30.Final-linux-x86_64.jar
libc:glibc 2.28 NPTL 2.28
OpenSSL: OpenSSL 1.0.2v-fips 5 May 2020
Providing below relevant sections of the crash dump file. Any inputs would be of great help.
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007fdb64c26a8c, pid=8969, tid=9065
#
# JRE version: OpenJDK Runtime Environment (Zulu11.39+16-SA) (11.0.7+10) (build 11.0.7+10-LTS)
# Java VM: OpenJDK 64-Bit Server VM (11.0.7+10-LTS, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x81a8c] cfree+0x1c
#
ost: Intel(R) Xeon(R) Gold 6142 CPU @ 2.60GHz, 4 cores, 15G, VMware Photon OS 3.0
--------------- T H R E A D ---------------
Current thread (0x00007fdaf8018000): JavaThread "nioEventLoopGroup-10-4" [_thread_in_native, id=9065, stack(0x00007fdab94f1000,0x00007fdab95f2000)]
Stack: [0x00007fdab94f1000,0x00007fdab95f2000], sp=0x00007fdab95efde8, free space=1019k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [libc.so.6+0x81a8c] cfree+0x1c
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 3701 io.netty.internal.tcnative.SSL.readFromSSL(JJI)I (0 bytes) @ 0x00007fdb4fbaa177 [0x00007fdb4fbaa120+0x0000000000000057]
J 10590 c2 io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap([Ljava/nio/ByteBuffer;II[Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (1438 bytes) @ 0x00007fdb5032c00c [0x00007fdb5032ba40+0x00000000000005cc]
J 6066 c2 io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(Lio/netty/handler/ssl/SslHandler;Lio/netty/buffer/ByteBuf;IILio/netty/buffer/ByteBuf;)Ljavax/net/ssl/SSLEngineResult; (134 bytes) @ 0x00007fdb4fd49d90 [0x00007fdb4fd499e0+0x00000000000003b0]
J 10485 c2 io.netty.handler.ssl.SslHandler.unwrap(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;II)I (629 bytes) @ 0x00007fdb5034d364 [0x00007fdb5034d080+0x00000000000002e4]
J 10202 c2 io.netty.handler.ssl.SslHandler.decode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (31 bytes) @ 0x00007fdb4fee5bc0 [0x00007fdb4fee58a0+0x0000000000000320]
J 10198 c2 io.netty.handler.codec.ByteToMessageDecoder.callDecode(Lio/netty/channel/ChannelHandlerContext;Lio/netty/buffer/ByteBuf;Ljava/util/List;)V (172 bytes) @ 0x00007fdb4fee92c8 [0x00007fdb4fee91e0+0x00000000000000e8]
J 6825 c2 io.netty.handler.codec.ByteToMessageDecoder.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (328 bytes) @ 0x00007fdb4fe2a200 [0x00007fdb4fe29d80+0x0000000000000480]
J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
J 7050 c2 io.netty.handler.timeout.IdleStateHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (42 bytes) @ 0x00007fdb4fe5eafc [0x00007fdb4fe5eaa0+0x000000000000005c]
J 8672 c2 io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Ljava/lang/Object;)Lio/netty/channel/ChannelHandlerContext; (12 bytes) @ 0x00007fdb500e9f28 [0x00007fdb500e9cc0+0x0000000000000268]
J 6504 c2 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (9 bytes) @ 0x00007fdb4fdd6964 [0x00007fdb4fdd6920+0x0000000000000044]
J 9935 c2 io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read()V (298 bytes) @ 0x00007fdb4fea0678 [0x00007fdb4fea0180+0x00000000000004f8]
J 9969 c2 io.netty.channel.nio.NioEventLoop.processSelectedKey(Ljava/nio/channels/SelectionKey;Lio/netty/channel/nio/AbstractNioChannel;)V (136 bytes) @ 0x00007fdb4fea9300 [0x00007fdb4fea9220+0x00000000000000e0]
J 10373% c2 io.netty.channel.nio.NioEventLoop.run()V (419 bytes) @ 0x00007fdb5033cef4 [0x00007fdb5033ca20+0x00000000000004d4]
j io.netty.util.concurrent.SingleThreadEventExecutor$4.run()V+44
j io.netty.util.internal.ThreadExecutorMap$2.run()V+11
j io.netty.util.concurrent.FastThreadLocalRunnable.run()V+4
j java.lang.Thread.run()V+11 java.base@11.0.7
v ~StubRoutines::call_stub
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000288
Top of Stack: (sp=0x00007fdab95efde8)
0x00007fdab95efde8: 00007fdaea98389a 00007fdb142ab520
0x00007fdab95efdf8: 00007fdaea7ed9fc 0000000000000001
0x00007fdab95efe08: 00007fdb5cf82540 0000000000000100
0x00007fdab95efe18: 00007fdab95effd0 00007fdb5cf82540
Instructions: (pc=0x00007fdb64c26a8c)
0x00007fdb64c2698c: 85 c0 48 89 c2 75 2c 4d 85 e4 74 27 0f 1f 84 00
0x00007fdb64c2699c: 00 00 00 00 4c 89 e7 48 89 ee e8 b5 c5 ff ff 48
0x00007fdb64c269ac: 89 ea 48 89 de 48 89 c7 49 89 c4 e8 24 ed ff ff
0x00007fdb64c269bc: 48 89 c2 4d 85 e4 0f 84 25 ff ff ff e9 f1 fe ff
0x00007fdb64c269cc: ff 0f 1f 00 48 8b 05 a1 04 13 00 31 d2 64 c7 00
0x00007fdb64c269dc: 16 00 00 00 e9 e9 fd ff ff 48 8d 0d 6c bd 0f 00
0x00007fdb64c269ec: 48 8d 35 13 7a 0f 00 48 8d 3d 4e bb 0f 00 ba e9
0x00007fdb64c269fc: 0c 00 00 e8 ec b4 ff ff 48 89 fb e9 6a fd ff ff
0x00007fdb64c26a0c: 0f 1f 40 00 48 83 ec 18 48 8b 05 cd 04 13 00 48
0x00007fdb64c26a1c: c7 00 00 00 00 00 8b 05 3c 08 13 00 85 c0 78 14
0x00007fdb64c26a2c: 48 8b 54 24 18 48 83 c4 18 e9 06 fd ff ff 66 0f
0x00007fdb64c26a3c: 1f 44 00 00 48 89 74 24 08 48 89 3c 24 e8 c2 bb
0x00007fdb64c26a4c: ff ff 48 8b 74 24 08 48 8b 3c 24 48 8b 54 24 18
0x00007fdb64c26a5c: 48 83 c4 18 e9 db fc ff ff 66 66 2e 0f 1f 84 00
0x00007fdb64c26a6c: 00 00 00 00 48 8b 05 89 04 13 00 48 8b 00 48 85
0x00007fdb64c26a7c: c0 0f 85 dd 00 00 00 48 85 ff 0f 84 ac 00 00 00
0x00007fdb64c26a8c: 48 8b 47 f8 48 8d 77 f0 a8 02 74 48 8b 15 1e 08
0x00007fdb64c26a9c: 13 00 85 d2 75 33 48 3b 05 e7 07 13 00 76 2a 48
0x00007fdb64c26aac: 3d 00 00 00 02 77 22 48 3b 35 7e 2e 13 00 0f 83
0x00007fdb64c26abc: b0 00 00 00 48 83 e0 f8 48 89 05 c5 07 13 00 48
0x00007fdb64c26acc: 01 c0 48 89 05 ab 07 13 00 48 89 f7 e9 43 b9 ff
0x00007fdb64c26adc: ff 0f 1f 00 48 89 c2 83 e2 04 48 83 3d ba 07 13
0x00007fdb64c26aec: 00 00 74 2b 48 8b 0d 91 02 13 00 48 85 d2 64 4c
0x00007fdb64c26afc: 8b 01 74 40 48 89 f7 48 81 e7 00 00 00 fc 48 8b
0x00007fdb64c26b0c: 3f 49 39 f8 75 1a 48 83 e0 f8 64 48 29 41 08 48
0x00007fdb64c26b1c: 85 d2 74 2c 48 89 f0 48 25 00 00 00 fc 48 8b 38
0x00007fdb64c26b2c: 31 d2 e9 0d cb ff ff 0f 1f 44 00 00 f3 c3 66 0f
0x00007fdb64c26b3c: 1f 44 00 00 48 8d 3d f9 10 13 00 49 39 f8 74 c6
0x00007fdb64c26b4c: 48 8d 3d ed 10 13 00 31 d2 e9 e6 ca ff ff 66 0f
0x00007fdb64c26b5c: 1f 44 00 00 48 8b 34 24 ff e0 66 2e 0f 1f 84 00
0x00007fdb64c26b6c: 00 00 00 00 48 3b 35 b9 2d 13 00 0f 83 43 ff ff
0x00007fdb64c26b7c: ff e9 53 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00
Stack slot to memory mapping:
stack at sp + 0 slots: 0x00007fdaea98389a: SSL_SESSION_free+0x000000000000012a in /lib/libssl.so.1.1 at 0x00007fdaea942000
stack at sp + 1 slots: 0x00007fdb142ab520 points into unknown readable memory: 70 f3 3f 14 db 7f 00 00
stack at sp + 2 slots: 0x00007fdaea7ed9fc: OPENSSL_LH_doall_arg+0x000000000000004c in /lib/libcrypto.so.1.1 at 0x00007fdaea67c000
stack at sp + 3 slots: 0x0000000000000001 is an unknown value
stack at sp + 4 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00
stack at sp + 5 slots: 0x0000000000000100 is an unknown value
stack at sp + 6 slots: 0x00007fdab95effd0 is pointing into the stack for thread: 0x00007fdaf8018000
stack at sp + 7 slots: 0x00007fdb5cf82540 points into unknown readable memory: c0 cc 9c ea da 7f 00 00
VM state:not at safepoint (normal execution)
VM Mutex/Monitor currently owned by a thread: None
Heap address: 0x0000000705400000, size: 4012 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
Compressed class space size: 1073741824 Address: 0x0000000800000000
Heap:
garbage-first heap total 311296K, used 166454K [0x0000000705400000, 0x0000000800000000)
region size 1024K, 132 young (135168K), 2 survivors (2048K)
Metaspace used 47712K, capacity 48362K, committed 48668K, reserved 1091584K
class space used 5432K, capacity 5678K, committed 5760K, reserved 1048576K
Metaspace:
Usage:
Non-class: 41.68 MB capacity, 41.29 MB (>99%) used, 317.92 KB ( <1%) free+waste, 86.06 KB ( <1%) overhead.
Class: 5.54 MB capacity, 5.31 MB ( 96%) used, 209.49 KB ( 4%) free+waste, 35.69 KB ( <1%) overhead.
Both: 47.23 MB capacity, 46.59 MB ( 99%) used, 527.41 KB ( 1%) free+waste, 121.75 KB ( <1%) overhead.
Virtual space:
Non-class space: 42.00 MB reserved, 41.90 MB (>99%) committed
Class space: 1.00 GB reserved, 5.62 MB ( <1%) committed
Both: 1.04 GB reserved, 47.53 MB ( 4%) committed
Chunk freelists:
Non-Class: 140.00 KB
Class: 0 bytes
Both: 140.00 KB
MaxMetaspaceSize: unlimited
CompressedClassSpaceSize: 1.00 GB
CodeHeap 'non-profiled nmethods': size=120032Kb used=10300Kb max_used=10300Kb free=109731Kb
bounds [0x00007fdb4fa32000, 0x00007fdb50452000, 0x00007fdb56f6a000]
CodeHeap 'profiled nmethods': size=120032Kb used=12721Kb max_used=19635Kb free=107310Kb
bounds [0x00007fdb484fa000, 0x00007fdb4985a000, 0x00007fdb4fa32000]
CodeHeap 'non-nmethods': size=5696Kb used=1238Kb max_used=1298Kb free=4457Kb
bounds [0x00007fdb47f6a000, 0x00007fdb481da000, 0x00007fdb484fa000]
total_blobs=17519 nmethods=8176 adapters=533
compilation: enabled
stopped_count=0, restarted_count=0
full_count=0
VM Arguments:
jvm_args: -Dlog4j.configuration=file:/............/conf/log4j.properties
java_command: /......./application-version.jar
java_class_path (initial): /......./application-version.jar
Launcher Type: SUN_STANDARD
[Global flags]
intx CICompilerCount = 3 {product} {ergonomic}
uint ConcGCThreads = 1 {product} {ergonomic}
uint G1ConcRefinementThreads = 4 {product} {ergonomic}
size_t G1HeapRegionSize = 1048576 {product} {ergonomic}
uintx GCDrainStackTargetSize = 64 {product} {ergonomic}
size_t InitialHeapSize = 264241152 {product} {ergonomic}
size_t MarkStackSize = 4194304 {product} {ergonomic}
size_t MaxHeapSize = 4206886912 {product} {ergonomic}
size_t MaxNewSize = 2523922432 {product} {ergonomic}
size_t MinHeapDeltaBytes = 1048576 {product} {ergonomic}
uintx NonNMethodCodeHeapSize = 5830732 {pd product} {ergonomic}
uintx NonProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
uintx ProfiledCodeHeapSize = 122913754 {pd product} {ergonomic}
uintx ReservedCodeCacheSize = 251658240 {pd product} {ergonomic}
bool SegmentedCodeCache = true {product} {ergonomic}
bool UseCompressedClassPointers = true {lp64_product} {ergonomic}
bool UseCompressedOops = true {lp64_product} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
Logging:
Log output configuration:
#0: stdout all=warning uptime,level,tags
#1: stderr all=off uptime,level,tags
Environment Variables:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
libc:glibc 2.28 NPTL 2.28