I have a 16 node cluster where every node has Spark and Cassandra installed. I am using dataframes so to utilize Spark MLlib also. So, I am wondering which of the following methods would be faster and less costly to filter a cassandra table based on a list of partition keys. Specifically, to be able to choose 4827 partition keys which contain ~2 billion rows from a cassandra table.
Spark offers the
- Join method (BroadcastHashJoin)
- filter.isin (which will pushdown predicates to cassandra, but I think it may put a big load on one node while also there may be missing data depending on the node it hits)
SCC offers the
- DirectJoin method (which for some reason in my case takes more time(~20mins) than BroadcastHashJoin(~7mins) to filter those 4827 partition keys.**)
**I have not used repartionbycassandrareplica with DirectJoin because it gives me an empty dataframe (asked here).