Which is best using separate kafka template / using same kafka template for different topic

Viewed 32

My application needs to send different records to different topics. My application is using the same Kafka cluster. Since the application uses the same Kafka cluster, creating one producer factory is sufficient(Let me know if I need more).

In my mind, I have two options.

  1. Using the same kafkaTemplate for both topics and calling the send method with the topic as below(Kindly assume I used spring default Kafka producer configurations). here we need to pass the topic for each call & we use the same Kafka template for multiple topics.
class ProducerService {
    @Autowired
    private KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate;

    public void send(String topic, GenericRecord key, GenericRecord value) {
        ListenableFuture<SendResult<GenericRecord, GenericRecord>> future = kafkaTemplate.send(topic, key, value);
    }
}
  1. Using different Kafka templates for different topics. I want to know Is this setup will increase the performance.

import org.apache.avro.generic.GenericRecord;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.core.ProducerFactory;

@Configuration
public class KafkaConfig {

    @Value("kafka.topic.first")
    private String firstTopic;
    @Value("kafka.topic.second")
    private String secondTopic;

    @Bean(name = "firstKafkaTemplate")
    public KafkaTemplate<GenericRecord, GenericRecord> firstKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
        KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
        kafkaTemplate.setDefaultTopic(firstTopic);
        return kafkaTemplate;
    }

    @Bean(name = "secondKafkaTemplate")
    public KafkaTemplate<GenericRecord, GenericRecord> secondKafkaTemplate(ProducerFactory<GenericRecord, GenericRecord> defaultKafkaProducerFactory) {
        KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
        kafkaTemplate.setDefaultTopic(secondTopic);
        return kafkaTemplate;
    }

}


class ProducerService {
    @Autowired
    @Qualifier("firstKafkaTemplate")
    private KafkaTemplate<GenericRecord, GenericRecord> firstTopicTemplate;

    @Autowired
    @Qualifier("secondKafkaTemplate")
    private KafkaTemplate<GenericRecord, GenericRecord> secondTopicTemplate;

    public void send(String topic, GenericRecord key, GenericRecord value) {
        ListenableFuture<SendResult<GenericRecord, GenericRecord>> future;
        if ("first".equalsIgnoreCase(topic)) {
            future = firstTopicTemplate.sendDefault(key, value);
        } else if ("second".equalsIgnoreCase(topic)) {
            future = secondTopicTemplate.sendDefault(key, value);
        } else {
            throw new RuntimeException("topic is not configured");
        }
    }
}

Internally, Kafka does batch processing and sends batches to Kafka by a separate thread. Which way is a better way to send records to gain performance? or there is no difference in performance?

0 Answers
Related