I was wondering if some of the JVM gurus out there can briefly explain the following error. What does it actually mean in technical terms and what are the sequences of events that can lead to this error?
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
This error may be induced by corrupted index which in turn becomes corrupted due to java.io.IOException: No space left on device
Look into your_kafka_home/logs/server.log
It all starts like this:
[2019-11-22 17:27:25,204] FATAL [Replica Manager on Broker 1]: Halting due to unrecoverable I/O error while handling produce request: (kafka.server.ReplicaManager)
kafka.common.KafkaStorageException: I/O exception in append to log 'xxx-response-1-0'
at kafka.log.Log.append(Log.scala:443)
at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:451)
Caused by: java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:82)
at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:159)
at kafka.log.LogSegment.append(LogSegment.scala:110)
at kafka.log.Log.append(Log.scala:429)
... 21 more
is followed by (possibly mutilple times):
[2019-11-22 17:28:00,395] WARN Found a corrupted index file due to requirement
failed: Corrupt index found, index file (/opt/kafka/kafka-logs/__consumer_offsets-
9/00000000000000000000.index) has non-zero size but the last offset is 0 which is no
larger than the base offset 0.}. deleting /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.timeindex, /opt/kafka/kafka-logs/__consumer_offsets-9
/00000000000000000000.index and rebuilding index... (kafka.log.Log)
and finally comes to :
[2019-11-22 17:28:02,025] FATAL [Kafka Server 1], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.lang.InternalError: **a fault occurred in a recent unsafe memory access operation in compiled Java code**
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
at kafka.log.TimeIndex.maybeAppend(TimeIndex.scala:107)
at kafka.log.LogSegment.$anonfun$recover$1(LogSegment.scala:238