Can I use Kafka for multiple independent consumers sequential reads?

noam cohen 0
Viewed 29

I have the following use case:

50 students write their own code which consumes a preloaded dataset, and they will repeat it many times. They all need to do the same task: read the data in order, and process it. The dataset is a time series containing 600 million messages, each message is about 1.3KB. Processing will probably be in Spark, but not mandatory. The dataset is fixed and ReadOnly.

The data should be read at "reasonable speed" > 30MB/sec for each consumer.

I was thinking of setting kafka cluster with 3+ brokers, 1 topic, and 50 partitions.

My issue with the above plan is that each student (== consumer) must read all the data, regardless of what other consumers do.

Is Kafka a good fit for this? If so, how?

What if I relax the requirement of reading the dataset in order? i.e. a consumer can read the 600M messages in any order. Is it correct that in this case each consumer will simply pull the full topic (starting with "earliest)?

An alternative is to set an HDFS storage (we use Azure so it's called Storage Account) and simply supply a mount point. However, I do not have control of the throughput in this case.

Throughput calculation:
let's say 25 consumers run concurrently, each reading at 30MB/s -> 750MB/s . Assuming data is read from disk, and disk rate is 50MB/s, I need to read concurrently from 750/50 = 15 disks. Does it mean I need to have 15 brokers? I did not see how one broker can allocate partitions to several disks attached to it.

similar posts:

Kafka topic partitions to Spark streaming

How does one Kafka consumer read from more than one partition?

(Spring) Kafka appears to consume newly produced messages out of order

Kafka architecture many partitions or many topics?

Is it possible to read from multiple partitions using Kafka Simple Consumer?

apache-kafka
2 Answers

Processing will probably be in Spark, but not mandatory

An alternative is to set an HDFS storage (we use Azure)

Spark can read from Azure Blob Storage, so I suggest you start with that first. You can easily scale up Spark executors in parallel for throughput.

If want to use Kafka, don't base consumption rate on disk speed alone, especially when Kafka can do zero-copy transfers. Use kafka-consumer-perf-test script to test how fast your consumers can go with one partition. Or, better, if your data has some key other than timestamp that you can order by, then use that.

It's not really clear if each "50 students" does the same processing on the data set, or some pre computations can be done, but if so, Kafka Streams KTables can be setup to aggregate some static statistics of the data, if it's all streamed though a topic, that way, you can distribute load for those queries, and not need 50 parallel consumers.

Otherwise, my first thought would be to simply use a TSDB like OpenTSDB, Timescale or Influx, maybe Druid . Which could also be used with Spark, or queried directly.

OneCricketeer0

If you are using Apache Spark 3.0+ there are ways around consumer per partition bound, as it can use more executor threads than partitions are, so it's mostly about how fast your network and disks are. Kafka stores latest offsets in memory, so probably for your use case most of reads will be from memory.

Desired minimum number of partitions to read from Kafka. By default, Spark has a 1-1 mapping of topicPartitions to Spark partitions consuming from Kafka. If you set this option to a value greater than your topicPartitions, Spark will divvy up large Kafka partitions to smaller pieces. Please note that this configuration is like a hint: the number of Spark tasks will be approximately minPartitions. It can be less or more depending on rounding errors or Kafka partitions that didn't receive any new data.

https://spark.apache.org/docs/3.0.1/structured-streaming-kafka-integration.html

themoah0
Related