I have a Clickhouse cluster with 6 nodes, configured with 2 shards and 3 replicas
- shard1: cdh01, cdh02, cdh03;
- shard2: cdh04, cdh05, cdh06;
I created a Kafka engine table and use a MATERIALIZED VIEW to write data to the local table with a ReplicatedReplacingMergeTree engine. All tables are created on each node with ON CLUSTER {cluster_name}.
Now I found all data are writen to the local table on nodes of shard1, and there is no data in the local table on nodes of shard2. And the amount of data on nodes of shard1 is seemed to be not correct.
I collect the debug log on node cdh06, and got logs like: logs.
There are several tables with similar settings created before and read data from different kafka-topics which are working correctly. But now when creating such tables, the problem above would happen.