How to configure Kafka to behave like a FiFo queue?

user_11532975 0
Viewed 7592

I am having a small problem and that is that I have to use kafka as a notification system to start flows of my application, everything is fine, but I would like to know if I can alter the order when there are unread messages in the queue.

Let me explain: When there are two or more messages in the topic, and the consumer reads them, read first, the last one that has entered, and I want it to read first, the first one that entered.

So I wanted to know if it is possible and if it is, that we have to change for it.

I am using the latest version of Spring Apache, and Kafka 2.5.0

ConsumerConfig:

@EnableKafka
@Configuration
public class KafkaConsumerConfig {

    private static final String BOOTSTRAP_ADDRESS = "";

    @Bean
    public ConsumerFactory<String, String> consumerFactory() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, BOOTSTRAP_ADDRESS);
        props.put(ConsumerConfig.GROUP_ID_CONFIG, "id");
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        return new DefaultKafkaConsumerFactory<>(props);
    }


    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory() {

        ConcurrentKafkaListenerContainerFactory<String, String> factory = new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory());
        return factory;
    }
}

KafkaConsumer:


@Component
public class KafkaConsumer {

    @KafkaListener(topics = "Great_Topic")
    private void listen(String msg) {
        System.out.println("I've received: " + msg);
    }
}

Example:

In my producer I have sent the following messages in this order:

  • Hello World 1
  • Hello World 2

Output in consumer:

  • Hello World 2
  • Hello World 1

I want the output to have the same order as the input.

apache-kafka spring java
1 Answers

When you consume from a topic you can opt to read from the beginning, and when you do so you get all the of the messages in the order in which they were produced into the partition.

If you have a single partition you will get strict FIFO behaviour.

If you have multiple partitions (as it looks like you have here) then you get FIFO within a partition, but across partitions it will be non-deterministic. Thus you could get the behaviour you show.

user_3506130
Related