Kafa + Structured Streaming : Read from topics in parallel

Viewed 111

I'm building a real-time data pipeline with the below flow

Source --> Kafka --> Structured Streaming (2.3) --> Sink

In doing so, I read from 3 topics.

Topic 1: sample_t1 (weekly) (written to Sink_Weekly) 
Topic 2: sample_t2 (once a day) (written to Sink_Daily)
Topic 3: sample_t3 (hourly) (Join with Sink_Daily and update Sink_Daily)

TL;DR

Current Approach:

At the moment, I'm reading all the topics as shown below with the intent that "no matter when data arrives, I will be ready to process it.. no matter which topic it is coming from".

 df = spark 
        .readStream 
        .format("kafka") 
        .option("kafka.bootstrap.servers", "localhost:9092") 
        .option("subscribePattern", "sample.*") 
        .option("failOnDataLoss", "false")
        .load()

query = df.writeStream
    .format("console")
    .foreach(new JoinWithSink(connection)) // Logic to perform joins as I get in the data
    .option("checkpointLocation", path/to/checkpoint/dir)
    .start()

query.awaitTermination()

But this approach got me thinking if this is the most optimal approach to handle topics with different frequencies.

Alternate Solution:

Another solution I have is to read each topic as a separate streaming query and then use stream-stream join to carry out the use case

df_weekly = spark 
        .readStream 
        .format("kafka") 
        .option("kafka.bootstrap.servers", "localhost:9092") 
        .option("subscribe", "sample_t1") 
        .option("failOnDataLoss", "false")
        .load()


 df_daily = spark 
        .readStream 
        .format("kafka") 
        .option("kafka.bootstrap.servers", "localhost:9092") 
        .option("subscribe", "sample_t2") 
        .option("failOnDataLoss", "false")
        .load()

 df_hourly = spark 
        .readStream 
        .format("kafka") 
        .option("kafka.bootstrap.servers", "localhost:9092") 
        .option("subscribe", "sample_t3") 
        .option("failOnDataLoss", "false")
        .load()

Use Case:

I want to be able to perform joins/lookup on the faster streaming query (Topic1) and update the sink accordingly.

Questions:

  1. Is it better to use subscribePattern and read all topics at once or is it better to split the topics into different streaming queries?
  2. How do I run multiple streaming queries in parallel? use case: run joins between topic1 and topic2 in parallel with joins between topic1 and topic3?
  3. In case of failure, how do I restart the streaming query by ignoring the corrupt record?
  4. Stop and restart the streaming query after x hours programmatically? I know I can add Triggers or use cron on a wrapper script.. is there a better way to stops and re-start the query?

Please let me know if you need more details, I will be happy to provide any additional detais.

0 Answers
Related