Advice on using 2 kafka consumers in python-kafka for 2 different topics in the same script and updating an object common to both of them

Long story short:

The question is how can I have 2 kafka consumers in the same python script, each of them consuming a different topic and updating the same data structure (ie: a pandas DataFrame).

Code is irrelevant for this question but may help in the answers section. I will try to describe the problem concisely.

TL'DR:

Description of the problem:

First topic holds historical data for stock symbols (let's call it t_history) and the other topic receives candlestick updates for each symbol (t_candlesticks).

I need to get the history first for each symbol (and still watch the topic) and create a structure from it (a dict with a dataframe for each symbol), while at the same time receive updated candlesticks (in the other topic) for some stocks and append them to the corresponding dataframe in the dict, dataframe that was defined based on the history from the first topic.

The loop for the historical topic never ends (which is expected, because I can add more symbols history to it), so how can I start the other consumer in a loop and have some processing without closing the historical loop and both updating the same object when needed?

Data:

The history is a big array of arrays that I parse and convert to a DataFrame with specific columns. Data is stored in the t_history topic with the key being the symbol (ie: TSLA, AAPL) and the value being the big array.

The realtime candlesticks are JSONs with a structure containing price and volume data {Open:100, Close:105, High:110, Low:95, Volume:12} and are stored in the t_candlesticks topic using the same key as above.

The common object is a dictionary with the keys being the symbols (TSLA) and the value being a dataframe that was created initially based on history and then updated with data coming from the candlestick topic.

Current options:

1. read with a poll the existing history, close the Consumer and then start the other Consumer for real-time data. CONs: no new history will ever be processed. PROs: simplest option and it works already

2. have a single consumer for both topics and when a message is received (based on the topic of the new message) I can perform the due processing (create the structure if it's history or append to it if it is a candlestick). PROs: simple, single-threaded, no concurrency issues for the common structure. CONs: If i receive a stock update first (before getting the history) there would be no structure in place to append it to (simply put: I need to read history first and then the updates). Another CON: being CPU bound, all the processing is done in a single thread/core not leveraging all the CPU cores or the processing power available.

3. Using threads, async or multiprocessing. For example: create a new thread/process for each consumer, maybe others for data processing. PROs: leverage all CPU power and all cores. CONs: I am not confident in the consistency of the object that would be shared across threads/processes (the dict with many dataframes); things may easily get out of control. And, since I am not very skilled in Python I have no idea how to achieve the best architecture for this.

The script is IO-bound for kafka consumption and CPU-bound for the processing of the real-time data. The history topic is rarely updated, usually once for each symbol in the beginning. The candlesticks topic is much more active.

Any options or any advice would be greatly appreciated. Code samples, architecture suggestions or even reading resources would also be beneficial.

Thanks in advance for taking the time to read the whole thing! :)