Getting KafkaConsumer is not safe for multi-threaded access error when I use CuratorFrameworkFactory.newClient()

Viewed 14238

I am receiving this error below:

java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access

when I include this in my code:

CuratorFrameworkFactory.newClient()

I don't understand what's causing the error. Any help?

Thanks.

2 Answers

here are some example to handle several topics thread-safe.

  1. One topic Consumer Per Thread, avoiding multi-threaded access error

    class Consumer implements Runnable {
        private volatile boolean exit = false;
    
        private String topic;
    
        Consumer(String topic){
            this.topic=topic;
        }
    
        @Override
        public void run() {
            KafkaConsumer<String, String> consumer = new KafkaConsumer<String,String>(props);
    
            try{
                consumer.subscribe(Collections.singletonList(topic));
    
                while (!exit) {
                    ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
    
                    for (ConsumerRecord<String, String> record : records){
                        System.out.print("Topic: " + record.topic() + ", ");
                        System.out.print("Partition: " + record.partition() + ",");
                        System.out.print("Key:" + record.key() + ", ");
                        System.out.println("Value: " + record.value() + ", ");
    
                        //do your stuff here
    
                    }
                }
            } catch (Exception e){
                e.printStackTrace();
            }finally {
                consumer.close();
            }
        }
        public void stop() {
           exit = true;
        }
     }
    
  2. One consumer with dynamic topic add, and multi-threaded access synchronized.

    class Consumer implements Runnable {
        private volatile boolean exit = false;
    
        private final List<String> topics;
        private final KafkaConsumer<String, String> consumer;
    
        Consumer(){
            topics = new ArrayList<>();
            consumer = new KafkaConsumer<String,String>(props);
        }
    
        public void addTopic(String newTopic){
             synchronized(this) {
                topics.add(newTopic);
                consumer.subscribe(topics);
            }
        }
    
        @Override
        public void run() {
            try{
                while (!exit) {
                    synchronized(this) {
                        ConsumerRecords<String, String> records = consumer.poll(Duration.ofSeconds(1));
                        for (ConsumerRecord<String, String> record : records){
                            System.out.print("Topic: " + record.topic() + ", ");
                            System.out.print("Partition: " + record.partition() + ",");
                            System.out.print("Key:" + record.key() + ", ");
                            System.out.println("Value: " + record.value() + ", ");
    
                            //do your stuff here
                        }
                    }
    
                    //IMPORTANT: sleep to avoid lock addTopic method
                    try { Thread.sleep(200); } catch (Exception ex) { }
                }
            } catch (Exception e){
                e.printStackTrace();
            }finally {
                consumer.close();
            }
        }
        public void stop() {
           exit = true;
        }
     }
    

I hope it helps

Related