Problem:
Setting up Multi datacenter using Docker Swarm. Each Docker Swarm at local DC is running with cassandra instances as 1/1+2 mode. There is a password connection between datacenters.
The seed nodes are DC-1:Node1, DC-2:Node1 (1+1) geo. DC-1:Node1, DC-1:Node3, DC-2:Node1, DC-2:Node3 in 1+2 mode of geo cluster ...
To discover the nodes, construct the topology between DC nodes via using cassandra storage port always expects bridge or host network. it does not work with OVERLAY network with PORT forwarding approach (Where it works for same DC with local network not across GEO sites).
It expect eithers host/bridge network, Otherwise it throws an exception as shown below
DEBUG [MessagingService-Outgoing-site-cassandra-A/15.29.8.10-Gossip] 2020-12-04 09:49:11,325
OutboundTcpConnection.java:546 - Unable to connect to site-cassandra-B/15.29.8.10
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_262]
at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_262]
at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_262]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:645) ~[na:1.8.0_262]
at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~
[apache-cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:434) [apache-
cassandra-3.11.4.jar:3.11.4]
at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:262) [apache-
cassandra-3.11.4.jar:3.11.4]
`
Workaround:
After we setup bridge network between geo-sites, it is able to discover and nodetool status shows two DC with proper cassandra instances with their replication configured % value.
I would like to know the reason of why cassandra is forcing to have
bridge or host based network why not with overlay base port forwarding
approach?
Thanks Suresh Perumal