java.lang.IllegalArgumentException: A consumerFactory is required

Viewed 20

This configuration existed before my changes:

@Configuration
public class KafkaConfiguration {

    @Value(value = "${kafka.bootstrapServers:XXX}")
    private String bootstrapServers;

    @Value(value = "${kafka.connect.url:XXX}")
    public String kafkaConnectUrl;

    @Bean
    public AdminClient adminClient() {
        Map<String, Object> configs = new HashMap<>();
        configs.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);

        return AdminClient.create(configs);
    }

    @Bean
    public WebProperties.Resources resources() {
        return new WebProperties.Resources();
    }

     @Bean
    public KafkaConnectClient connectClient() {
        return new KafkaConnectClient(new org.sourcelab.kafka.connect.apiclient.Configuration(kafkaConnectUrl));
    }
}

I've setup the app so that I can use KafkaTemaplate:

application.yaml

    spring:
      profiles:
        active: dev
    
      kafka:
        bootstrap-servers: XXX
        properties:
          schema.registry.url: XXX
        producer:
          key-serializer: org.apache.kafka.common.serialization.StringSerializer
          value-serializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer
        consumer:
          group-id: kafka-management-service-group-id
          auto-offset-reset: earliest
          key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
          value-deserializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer

I can correctly write and read messages using KafkaTemplate from a topic with 1 partition that I created manually:

@GetMapping("/loadData")
public void loadData() {
    IntStream.range(0, 1000000).forEach(key ->
        {
            final General general = General.newBuilder()
                .setUuid(StringValue.newBuilder().setValue(String.valueOf(key))
                    .build()).build();
            template.send("test",
                String.valueOf(key),
                ScheduledJob.newBuilder().setGeneral(general).build()
            );
        }
    );
}

@GetMapping("/readData")
public void readData() {
    Collection<TopicPartitionOffset> topicPartitionOffsets = new ArrayList<>();
    final TopicPartitionOffset topicPartitionOffset = new TopicPartitionOffset("test", 0, SeekPosition.BEGINNING);
    topicPartitionOffsets.add(topicPartitionOffset);
    final Iterator<ConsumerRecord<String, ScheduledJob>> iterator = template.receive(topicPartitionOffsets).iterator();
    while (iterator.hasNext()) {
        final ConsumerRecord<String, ScheduledJob> next = iterator.next();
        System.out.println("Key = "+next.key());
        System.out.println("Value = "+next.value());
        System.out.println();
    }
}

Current task: I'd like to read all messages of a topic from a specific partition X (therefore I need the number of partitions), from a specific offet Y to a specific offset Z; and ignore the rest.

I've read that I can do something along these lines to get the partition number for a specific topic:

@Autowired
private final ConsumerFactory<Object, Object> consumerFactory;

public String[] partitions(String topic) {
    try (Consumer<Object, Object> consumer = consumerFactory.createConsumer()) {
        return consumer.partitionsFor(topic).stream()
            .map(pi -> "" + pi.partition())
            .toArray(String[]::new);
    }
}

Problem: Despite the fact that I can read and write messages, a ConsumerFactory isn't created automatically.

Context: Our frontend needs to support paging, multi-column order and filtering. The source are Kafka messages from a generic topic with any number of partitions. Maybe this isn't the best way to do it, therefore I'm also open for suggestions

1 Answers

I don't see how it is possible to receive records without adding a consumer factory to the template; Boot does not do that. See KafkaTemplate.oneOnly(), which is called by the receive methods...

private Properties oneOnly() {
    Assert.notNull(this.consumerFactory, "A consumerFactory is required");
    Properties props = new Properties();
    props.setProperty(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, "1");
    return props;
}

Boot auto configures a consumer factory, but it does not add it to the template because using template to receive is an uncommon use case.

/**
 * Set a consumer factory for receive operations.
 * @param consumerFactory the consumer factory.
 * @since 2.8
 */
public void setConsumerFactory(ConsumerFactory<K, V> consumerFactory) {
    this.consumerFactory = consumerFactory;
}
Related