Ec2MultiRegionSnitch was made I assume because a few years back AWS did not support VPC peering. With the current VPC peering capabilities of AWS, we can peer two regions and set up cassandra in both of them with Ec2Snitch.
As long as they can communicate cassandra will recognize the two datacenters as different and all should be well.
Or so I thought. I saw only one case of this throughout the internet and that was this issue: https://issues.apache.org/jira/browse/CASSANDRA-15337 Who claims he has done it.
In my case whenever I have 2 datacenters one in us and one in eu with 4 nodes each and a RF=3.
Whenever I insert data with consistency of LOCAL_QUORUM hitting a EU coordinator all is well and the data is inserted in the correct nodes ( EU nodes ) and later replicated over to the US ones. But when I insert data in the US one from the tracing I see that we hit the EU one.
Does anyone have any experience with this?