Low Performance of Spark Report when Spark and Cassandra on separate Docker Containers

user_12988036 0
Viewed 63

I am using Spark 3.0.1, spark cassandra connector and cassandra on Kubernetes.

I was using spark cassandra connector repartitionByCassandraReplica API to get the data locality feature of Spark Partition with Cassandra and then joinWithCassandraTable. But this repartitioning is failed as cassandra data is not local for Spark container. Because of which joinWithCassandraTable performance is getting very low.

Is there any other way to get the good performance for joinWithCassandraTable.

spark-cassandra-connector apache-spark kubernetes cassandra
1 Answers

As you already know, calling repartitionByCassandraReplica is pointless when Spark and Cassandra are not co-located on the same machines.

To maximise the throughput of your Cassandra cluster are:

  • allocate at least 4 cores to each C* pod (8 cores is recommended)
  • allocate at least 16GB of RAM to each C* pod (24-30GB is recommended)
  • allocate at least 8GB of memory to the heap (16GB is recommended)
  • allocate volumes which have at least 5K IOPS

Once you've configured these recommendations, increase the capacity of your cluster by adding more C* pods. Cheers!

user_42695350
Related