Hazelcast : IllegalArgumentException: Target is this node

Viewed 691

I am running two instances of spring boot application on openshift and both are joined to hazelcast cluster. However, I can see continuously see below error message in logs.

{"timestamp":"2021-06-01T20:29:48.775+10:00","app":"my-protected-application","logLevel":"INFO","thread":"hz.peaceful_pike.priority-generic-operation.thread-0","eventSource":"com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp","message":"[10.1.28.31]:5701 [spring-session-cluster] [4.2] Removing null, since it thinks it's already split from this cluster and looking to merge."}
{"timestamp":"2021-06-01T20:29:48.776+10:00","app":"my-protected-application","logLevel":"ERROR","thread":"hz.peaceful_pike.priority-generic-operation.thread-0","eventSource":"com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp","message":"[10.1.28.31]:5701 [spring-session-cluster] [4.2] Target is this node! -> [10.1.28.31]:5701","stack_trace":"<#d3566be0> j.l.IllegalArgumentException: Target is this node! -> [10.1.28.31]:5701\n\tat c.h.s.i.o.i.OutboundResponseHandler.checkTarget(OutboundResponseHandler.java:226)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendNormalResponse(OutboundResponseHandler.java:125)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:88)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.call(OperationRunnerImpl.java:282)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:248)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n"}
{"timestamp":"2021-06-01T20:29:48.776+10:00","app":"my-protected-application","logLevel":"WARN","thread":"hz.peaceful_pike.priority-generic-operation.thread-0","eventSource":"com.hazelcast.spi.impl.operationservice.impl.OperationRunnerImpl","message":"[10.1.28.31]:5701 [spring-session-cluster] [4.2] While sending op error... op: com.hazelcast.internal.cluster.impl.operations.SplitBrainMergeValidationOp{serviceName='hz:core:clusterService', identityHash=127643227, partitionId=-1, replicaIndex=0, callId=1046044, invocationTime=1622543389369 (2021-06-01 20:29:49.369), waitTimeout=-1, callTimeout=60000, tenantControl=com.hazelcast.spi.impl.tenantcontrol.NoopTenantControl@0}, error: java.lang.IllegalArgumentException: Target is this node! -> [10.1.28.31]:5701","stack_trace":"<#dbcc9949> j.l.IllegalArgumentException: Target is this node! -> [10.1.28.31]:5701, response: ErrorResponse{callId=1046044, urgent=true, cause=java.lang.IllegalArgumentException: Target is this node! -> [10.1.28.31]:5701}\n\tat c.h.s.i.o.i.OutboundResponseHandler.send(OutboundResponseHandler.java:113)\n\tat c.h.s.i.o.i.OutboundResponseHandler.sendResponse(OutboundResponseHandler.java:96)\n\tat c.h.s.i.o.Operation.sendResponse(Operation.java:475)\n\tat c.h.s.i.o.i.OperationRunnerImpl.sendResponseAfterOperationError(OperationRunnerImpl.java:425)\n\tat c.h.s.i.o.i.OperationRunnerImpl.handleOperationError(OperationRunnerImpl.java:419)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:253)\n\tat c.h.s.i.o.i.OperationRunnerImpl.run(OperationRunnerImpl.java:469)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:197)\n\tat c.h.s.i.o.i.OperationThread.process(OperationThread.java:137)\n\tat c.h.s.i.o.i.OperationThread.executeRun(OperationThread.java:123)\n\tat c.h.i.u.e.HazelcastManagedThread.run(HazelcastManagedThread.java:102)\n"}
{"timestamp":"2021-06-01T20:29:49.779+10:00","app":"my-protected-application","logLevel":"INFO","thread":"hz.peaceful_pike.InvocationMonitorThread","eventSource":"com.hazelcast.spi.impl.operationservice.impl.InvocationMonitor","message":"[10.1.28.31]:5701 [spring-session-cluster] [4.2] Invocations:2 timeouts:1 backup-timeouts:0"}

what does this log mean and what is it's significance.

1 Answers

Hazelcast for OpenShift

Check if you use StatefulSet (not Deployment). In the case of DNS Lookup discovery, using Deployment may cause Hazelcast to start in the split-brain mode. Also you may want to increase the value of the service-dns-timeout parameter.

Related