LEAK DETECTED: was not released before the reference was garbage collected

user_3646265 0
Viewed 548

I am getting the following errors in Cassandra 3.11.5

LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@356cb07f) to class org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1709177197:Memory@[7e7a3b64b910..7e7a3b96b910) was not released before the reference was garbage collected


ERROR [CompactionExecutor:3763] 2020-11-02 02:34:53,316  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3763,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
        ... 24 common frames omitted
WARN  [GossipTasks:1] 2020-11-02 02:34:53,302  FailureDetector.java:278 - Not marking nodes down due to local pause of 7468254535 > 5000000000
ERROR [Reference-Reaper] 2020-11-02 02:34:53,208  Ref.java:229 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@33fef449) to class org.apache.cassandra.io.util.FileHandle$Cleanup@1098210945:/data/cassandra/data/elvisevallogks/decision_detail-7351f6205f3811ea922ce1efeeba3e49/md-3695-big-Index.db was not released before the reference was garbage collected
ERROR [CompactionExecutor:3762] 2020-11-02 02:34:53,208  CassandraDaemon.java:228 - Exception in thread Thread[CompactionExecutor:3762,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:180) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:135) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:268) ~[apache-cassandra-3.11.5.jar:3.11.5]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_60]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_60]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_60]
Caused by: java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_60]
        at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153) ~[apache-cassandra-3.11.5.jar:3.11.5]
        ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_60]
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_60]
        ... 24 common frames omitted
INFO  [Service Thread] 2020-11-02 02:34:53,877  StatusLogger.java:51 - MemtablePostFlush                 0         0           3596         0                 0

These are the settings

# Cassandra-specfic parameters
# add
cassandra        soft    memlock         unlimited
cassandra        hard    memlock         unlimited
cassandra        soft    nproc           65535
#change
*                soft    nofile          655350
*                hard    nofile          655350

For cassandra process

cat /proc/25586/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    0                    bytes
Max resident set          unlimited            unlimited            bytes
Max processes             65535                692334               processes
Max open files            655350               655350               files
Max locked memory         unlimited            unlimited            bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       692334               692334               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

I searched online and the recommended setting is to set vm.max_map_count 1048575 and I verified that the value is setup correctly. The heap size is set to 32 GB and the machine has 171 GB of RAM.

sysctl vm.max_map_count
vm.max_map_count = 1048575

The node doesn't crash but I get random timeout when I try to connect.

cassandra-3.0 cassandra
1 Answers

Probably you might need to check and enable disk_access_mode from the default "Standard" to "mmap_index_only", it avoids page faults by using map only index files. the page faults cause memory leaks some times, and it is advisable to hardcode the heap to particular size based on your cluster utilization.

The problem is that when the heap defaults to 32 or higher, you loose compression of a lot of objects in the heap and it won't perform as well, so those two changes would be good together when you get an opportunity for a rolling restart.

thanks, Ganesh

user_145866330
Related