Continuously consume kafka message with Kafka Python

I want to build a processor to transform one Kafka json messages into several dataframes. I want to process 100,000 messages in batch or in 10 second intervals.

This is what I have:

    datastruct = []
    lead = 1
    for msg in consumer:
        start_time = time.time()
        record = json.loads(msg.value)
        datastruct.append(record)
        current_time = time.time()
        elapsed_time = current_time-start_time
        lead = lead + 1
        if(len(datastruct) >= 100000 or elapsed_time > 5):
            consumer.pause()
            *** transformation here ***
            datastruct.clear()
            consumer.resume()
    if consumer is not None:
        consumer.close()

Let say I have about 699,000 message at first.

My goal is to continue to check if there are any new message for each interval, but it stops if there are no messages in my kafka and not wait until new messages come. And it was only processed 600,000 messages, and left the 99,000 messages behind.

Any help is appreciated.