Cassandra write query timeout out after PT2S

user_1228785 0
Viewed 13273

I have cassandra monolithic application where I want to write at high rate reading some payloads from queue. Cassandra cluster has 3 nodes . When i start processing large number of messages in parallel(by spawning threads) I get below exceptions

java.util.concurrent.ExecutionException: com.datastax.oss.driver.api.core.DriverTimeoutException: Query timed out after PT2S

I am creating CQLsession as bean

return CqlSession.builder().addContactPoints(contactPoints)
            /*.addContactPoint(new InetSocketAddress("localhost", 9042))*/
            .withConfigLoader(new DefaultDriverConfigLoader()).withLocalDatacenter("datacenter1")
            .addTypeCodecs(new CustomDateCodec())
            .withKeyspace("dev").build();

I am injecting this CqlSession into my mapper and other classes to run queries

In my datastax driver i have given ip of 3 nodes as contact points Is there any tuning I need to do in CQLsession creation/ or my cassandra nodes so that they can take is writes at high concurrency ? Also How many writes can I do in parallel ?

All are update statement without any if condition only on primary key

datastax-java-driver cassandra
1 Answers

The timeout you're seeing is a result of your app overloading the cluster, effectively doing a DDoS attack.

PT2S is the 2-second write timeout. There will come a point when the commitlog disks can only take so much write IO. If you're seeing dropped mutations in the logs or nodetool tpstats, that's confirmation that the commitlog can't keep up with the writes.

If your cluster can sustain 10K writes/sec but your app is doing 20K writes then you need to double the size of your cluster (add more nodes) to support the throughput requirements. Cheers!

user_42695350
Related