Spring Kafka transaction - No transaction is in process, run the template operation within the scope of a template.executeInTransaction()

user_7200088 0
Viewed 2666

I'm trying to publish messages to Kafka from a transaction using KafkaTemplate:

@Autowired
KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate;

@Transactional
@RabbitListener(queues = "queueName")
void input(final List<Message> messages) {
     for (Message msg : messages) {
          PublishRequest request = prepareRequest(msg);
          kafkaTemplate.sendDefault(request.getKey(), reguest.getValue());
     }
     transactionalDatabaseInserts();
}

But when I do it I'm getting this exception:

Caused by: java.lang.IllegalStateException: No transaction is in process; possible solutions: run the template operation within the scope of a template.executeInTransaction() operation, start a transaction with @Transactional before invoking the template method, run in a transaction started by a listener container when consuming a record

Configs for KafkaTemplate:

@EnableTransactionManagement
@Configuration
public class KafkaConfig{
    @Bean
    KafkaTransactionManager<GenericRecord, GenericRecord> kafkaTransactionManager(final ProducerFactory<GenericRecord, GenericRecord> producerFactory) {
        return new KafkaTransactionManager<>(producerFactory);
    }

    @Bean
    KafkaTemplate<GenericRecord, GenericRecord> kafkaTemplate(final ProducerFactory<GenericRecord, GenericRecord> producerFactory) {
        return new KafkaTemplate<>(producerFactory);
    }
}

In my application.yaml I have included:

spring.kafka.producer.transaction-id-prefix: tx-

I want my method to work with @Transactional and not kafkaTemplate.executeInTransaction(). Why am I getting that exception?

spring-kafka apache-kafka spring-boot spring java
1 Answers

You must have something mis-configured - this works as expected...

@SpringBootApplication
@EnableTransactionManagement
public class So63596919Application {

    public static void main(String[] args) {
        SpringApplication.run(So63596919Application.class, args);
    }

    @Autowired
    private KafkaTemplate<String, String> template;

    private final CountDownLatch latch = new CountDownLatch(1);

    @Transactional
    @RabbitListener(queues = "so63596919")
    public void listen(List<String> in) throws InterruptedException {
        System.out.println(in);
        in.forEach(str -> this.template.send("so63596919", str));
        System.out.println("Hit enter to exit listener and commit transaction");
        this.latch.await();
    }

    @KafkaListener(id = "so63596919", topics = "so63596919")
    public void listen(String in) {
        System.out.println(in);
    }

    @Bean
    public Queue queue() {
        return new Queue("so63596919");
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so63596919").partitions(1).replicas(1).build();
    }

    @Bean
    public ApplicationRunner runner(RabbitTemplate template, AbstractRabbitListenerContainerFactory<?> factory) {
        factory.setBatchListener(true);
        factory.setContainerCustomizer(container -> {
                ((SimpleMessageListenerContainer) container).setConsumerBatchEnabled(true);
                container.setDeBatchingEnabled(true);
        });
        return args -> {
            template.convertAndSend("so63596919", "foo");
            template.convertAndSend("so63596919", "bar");
            System.in.read();
            this.latch.countDown();
        };
    }

}

spring.kafka.producer.transaction-id-prefix: tx-
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.properties.isolation.level=read_committed

spring.rabbitmq.listener.simple.batch-size=2

If you can trim down your project to a small example like this, I can take a look to see what's wrong.

user_12407630
Related