We are using Spring rabbitmq for out project. We are currently testing different failover scenario to prevent any further exception over Production.
We are using following properties: listener: type: direct direct: acknowledge-mode: manual
What we face is that When there is any network failure, we got following logs:
INFO [SimpleAsyncTaskExecutor-1][2018-10-22 15:37:55,726] - org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:463) - Created new connection: rabbitConnectionFactory#4c05df3b:0/SimpleConnection@2d361383 [delegate=amqp://workflow@192.168.156.53:5671/, localPort= 49445]
INFO [SimpleAsyncTaskExecutor-1][2018-10-22 15:37:55,789] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:613) - SimpleConsumer [queue=vishalr_dev.1482820471007_workflow, consumerTag=amq.ctag-TNSIb9kI1SxdO8cXtfOgrg identity=7155ef26] started
WARN [AMQP Connection 192.168.156.53:5671][2018-10-22 15:38:08,830] - com.rabbitmq.client.impl.ForgivingExceptionHandler.log(ForgivingExceptionHandler.java:115) - An unexpected connection driver error occured (Exception message: Connection reset)
ERROR [AMQP Connection 192.168.156.53:5671][2018-10-22 15:38:08,836] - org.springframework.amqp.rabbit.connection.CachingConnectionFactory$DefaultChannelCloseLogger.log(CachingConnectionFactory.java:1302) - Channel shutdown: connection error
ERROR [null-consumerMonitor-1][2018-10-22 15:38:12,285] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$null$3(DirectMessageListenerContainer.java:397) - Consumer canceled - channel closed SimpleConsumer [queue=vishalr_dev.1482820471007_workflow, consumerTag=amq.ctag-TNSIb9kI1SxdO8cXtfOgrg identity=7155ef26]
INFO [null-consumerMonitor-1][2018-10-22 15:38:17,282] - org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:447) - Attempting to connect to: [montu.local.com:5671]
ERROR [null-consumerMonitor-1][2018-10-22 15:38:17,290] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$4(DirectMessageListenerContainer.java:416) - Cannot connect to server
org.springframework.amqp.AmqpIOException: java.net.SocketException: Network is unreachable: connect
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:476) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:614) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createBareChannel(CachingConnectionFactory.java:564) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getCachedChannelProxy(CachingConnectionFactory.java:538) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.getChannel(CachingConnectionFactory.java:520) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.access$1500(CachingConnectionFactory.java:94) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$ChannelCachingConnectionProxy.createChannel(CachingConnectionFactory.java:1171) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:569) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$4(DirectMessageListenerContainer.java:413) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_141]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_141]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_141]
Caused by: java.net.SocketException: Network is unreachable: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_141]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_141]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_141]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_141]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_141]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_141]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_141]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_141]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) ~[na:1.8.0_141]
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60) ~[amqp-client-5.1.2.jar:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:955) ~[amqp-client-5.1.2.jar:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:907) ~[amqp-client-5.1.2.jar:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:847) ~[amqp-client-5.1.2.jar:5.1.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:449) ~[spring-rabbit-2.0.4.RELEASE.jar:2.0.4.RELEASE]
... 16 common frames omitted
After above log, it is not retrying to connect with consumer.
However we debug into spring jar itself and found that for org.springframework.amqp.AmqpIOException, it is not adding consumer for restart and it could be issue behind it.
Any idea , how we can resolve our network faiover scenario with this.
Here network failover means we have put firewall between rabbitmq server and consumer as a first test, as a second we have disable network adapter from consumer side.
UPDATE : 1 =============================================================================
Reason behind why we are testing this type of network failur is that, One day before we have some infrastucture issue, Due to which rabbitmq VM was down for 30min.
But when rabbitmq VM is running up again, All consumers was not connectted automatically as it should. We tried to regenerate this in our local but couldn't. Here is log from instance where consumers was not able to auto connect.
INFO [SimpleAsyncTaskExecutor-1][2018-10-22 10:15:51,142] - org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:447) - Attempting to connect to: [RABBITMQAIND1.hqdev.india:5671, RABBITMQBIND1.hqdev.india:5671]
ERROR [SimpleAsyncTaskExecutor-1][2018-10-22 10:15:51,142] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:459) - Error creating consumer; retrying in 5000
org.springframework.amqp.AmqpConnectException: java.net.ConnectException: Connection refused: connect
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:62) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:476) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:614) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:553) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consumeFromQueue(DirectMessageListenerContainer.java:535) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:446) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: java.net.ConnectException: Connection refused: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) ~[na:1.8.0_152]
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) ~[na:1.8.0_152]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_152]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_152]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_152]
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) ~[na:1.8.0_152]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_152]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_152]
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) ~[na:1.8.0_152]
at com.rabbitmq.client.impl.SocketFrameHandlerFactory.create(SocketFrameHandlerFactory.java:60) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:955) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:907) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:847) ~[amqp-client-5.1.2.jar!/:5.1.2]
at org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:449) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
... 5 common frames omitted
INFO [SimpleAsyncTaskExecutor-1][2018-10-22 10:15:54,472] - org.springframework.amqp.rabbit.connection.AbstractConnectionFactory.createBareConnection(AbstractConnectionFactory.java:463) - Created new connection: rabbitConnectionFactory#40ae8c5:85/SimpleConnection@54d6cde2 [delegate=amqp://workflow@192.168.154.191:5671/, localPort= 50085]
ERROR [SimpleAsyncTaskExecutor-1][2018-10-22 10:16:07,314] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:459) - Error creating consumer; retrying in 5000
org.springframework.amqp.AmqpConnectException: org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:557) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consumeFromQueue(DirectMessageListenerContainer.java:535) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:446) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: org.springframework.amqp.AmqpIOException: java.io.IOException
at org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:71) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.RabbitAccessor.convertRabbitAccessException(RabbitAccessor.java:113) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1834) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1771) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.execute(RabbitTemplate.java:1752) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitAdmin.initialize(RabbitAdmin.java:540) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$null$9(RabbitAdmin.java:453) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) ~[spring-retry-1.2.2.RELEASE.jar!/:na]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:164) ~[spring-retry-1.2.2.RELEASE.jar!/:na]
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$afterPropertiesSet$10(RabbitAdmin.java:452) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CompositeConnectionListener.onCreate(CompositeConnectionListener.java:36) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory.createConnection(CachingConnectionFactory.java:620) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:553) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
... 3 common frames omitted
Caused by: java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:947) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclare(ChannelN.java:52) ~[amqp-client-5.1.2.jar!/:5.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:991) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at com.sun.proxy.$Proxy180.queueDeclare(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.core.RabbitAdmin.declareQueues(RabbitAdmin.java:608) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitAdmin.lambda$initialize$11(RabbitAdmin.java:542) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.core.RabbitTemplate.doExecute(RabbitTemplate.java:1828) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
... 13 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@RABBITMQAIND1' of durable queue 'collabscrum1.1454137672227_workflow' in vhost '/' is down or inaccessible, class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:288) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:138) ~[amqp-client-5.1.2.jar!/:5.1.2]
... 24 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@RABBITMQAIND1' of durable queue 'collabscrum1.1454137672227_workflow' in vhost '/' is down or inaccessible, class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:504) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:643) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:581) ~[amqp-client-5.1.2.jar!/:5.1.2]
... 1 common frames omitted
ERROR [SimpleAsyncTaskExecutor-1][2018-10-22 10:16:07,345] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:593) - Queue not present, scheduling consumer SimpleConsumer [queue=neha_dev_workflow, consumerTag=null identity=5e8ed89c] for restart
java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:991) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.1.2.jar!/:5.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:991) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at com.sun.proxy.$Proxy180.queueDeclarePassive(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:572) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consumeFromQueue(DirectMessageListenerContainer.java:535) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:446) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@RABBITMQAIND1' of durable queue 'neha_dev_workflow' in vhost '/' is down or inaccessible, class-id=50, method-id=10)
at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:66) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.privateRpc(AMQChannel.java:288) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:138) ~[amqp-client-5.1.2.jar!/:5.1.2]
... 12 common frames omitted
Caused by: com.rabbitmq.client.ShutdownSignalException: channel error; protocol method: #method<channel.close>(reply-code=404, reply-text=NOT_FOUND - home node 'rabbit@RABBITMQAIND1' of durable queue 'neha_dev_workflow' in vhost '/' is down or inaccessible, class-id=50, method-id=10)
at com.rabbitmq.client.impl.ChannelN.asyncShutdown(ChannelN.java:504) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.processAsync(ChannelN.java:346) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:178) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:643) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:581) ~[amqp-client-5.1.2.jar!/:5.1.2]
... 1 common frames omitted
ERROR [SimpleAsyncTaskExecutor-1][2018-10-22 10:16:07,345] - org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:593) - Queue not present, scheduling consumer SimpleConsumer [queue=autoind9.1526118175953_workflow, consumerTag=null identity=6465a7f8] for restart
java.io.IOException: null
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:126) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:122) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.AMQChannel.exnWrappingRpc(AMQChannel.java:144) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:991) ~[amqp-client-5.1.2.jar!/:5.1.2]
at com.rabbitmq.client.impl.ChannelN.queueDeclarePassive(ChannelN.java:52) ~[amqp-client-5.1.2.jar!/:5.1.2]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_152]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_152]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_152]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_152]
at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:991) ~[spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at com.sun.proxy.$Proxy180.queueDeclarePassive(Unknown Source) ~[na:na]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.doConsumeFromQueue(DirectMessageListenerContainer.java:572) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.consumeFromQueue(DirectMessageListenerContainer.java:535) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at org.springframework.amqp.rabbit.listener.DirectMessageListenerContainer.lambda$actualStart$5(DirectMessageListenerContainer.java:446) [spring-rabbit-2.0.4.RELEASE.jar!/:2.0.4.RELEASE]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_152]
Is this log more useful to trace issue ?