Python threading - runtime error when using thread in a loop - any better solution?

Ahsan Iqbal 0
Viewed 38

I am designing a trading strategy to work with binance API as my research project. The following is the basic scheme of program

  1. The program needs to wait to receive a line of data (kline) from the API. The time needs to be different for different pairs and at different time of the day (which is set in a separate trading logic file). In short, the waiting time is not fixed (hence I am using threading.Condition() with .wait() and .notify() functions)
  2. Once the data is available, condition.notify() comes into action and technical analysis is performed on the line received
  3. Because I want the program to run forever, the thread is started in a while loop
  4. However, during my testing phase, after receiving around 12000 lines of data, the script gave me the following error

File "/usr/lib/python3.8/threading.py", line 852, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

I am thinking the system ran out of memory because it is creating a new thread during every iteration of the loop. Is there a better way to make the program wait to receive data? or to better manage the thread responsible for waiting?

During the testing phase, I am reading the data from csv files line by line.

tc_kline_received = threading.Condition()

for file in list_of_files:
    with open(file, "r") as csv_file:
        file_reader = reader(csv_file)
        for line in file_reader:
            btc_busd_5_min.append(line)

The following function pops one line from the list and stores in a separate variable. It is also used to for the thread responsible for waiting

def collect_kline():
global btc_busd_5_min
global kline_btc_busd_5_min
global klines_dict
kline_btc_busd_5_min = btc_busd_5_min.pop(0)
klines_dict["btc_busd_5_min"] = kline_btc_busd_5_min
with tc_kline_received:
    tc_kline_received.notify()

Below is the never ending while loop.

while True:
    with tc_kline_received:
        t_collect_kline = threading.Thread(target=collect_kline)
        t_collect_kline.start()
        tc_kline_received.wait()
        t_collect_kline.join()   
    insert_kline_to_db(klines_dict)
    create_ta_db(klines_dict)

The last two functions are to create a sqlite db with technical analysis.

runtime-error multithreading python
1 Answers

I am thinking the system ran out of memory because it is creating a new thread during every iteration of the loop.

The loop creates a new thread on every iteration, but it cannot get past the t_collect_kline.join() line until the new thread is finished, so there will never be more than one "collect_kline()" thread running at any given moment in time. If your program is running out of memory, it's not because of too many threads.

Unfortunately, I don't know what half of the lines in that program actually do, so I can't say what might be using up a lot of memory.

About those threads though...

...Each time your main loop creates a new thread, it then does nothing until the thread is finished. That doesn't seem useful. The whole point of creating a new thread is that the new thread can do one thing, while the original thread does some other thing. But, your original thread does nothing else. It never makes any sense to create a new thread if the very next thing you do is wait for it to end.

You could make your program smaller, simpler, and probably faster if you'd just do this in your main loop instead:

while True:
    global btc_busd_5_min
    global klines_dict
    klines_dict["btc_busd_5_min"] = btc_busd_5_min.pop(0)
    insert_kline_to_db(klines_dict)
    create_ta_db(klines_dict)

But like I said, if you've got an out-of-memory problem, then this change isn't going to fix it. It will only change when and where the out-of-memory condition gets reported.

Solomon Slow0
Related