Unable to catch one exception after another in MainThread

Viewed 31

To quickly explain how this code works, we have an object called Proxy, we have a queue with a lot of these objects, and they are being multithreaded to check if you can succesfuly do a request with its credentials.

The problem comes in the manner that, I get an item from the queue in the main thread, having a timeout on the get() queue method (which value is a couple seconds more than the timeout that we have set on the Proxy object class att for the request). This happens in a while True with a try catch which is catching both the timeout exception and KeyboardInterrupt, however, I am unable to catch KeyboardInterrupt in some cases, which I will explain below

import concurrent.futures
import queue
import threading
import requests


lock_ = threading.Lock()


class Proxy:
    timeout = 3

    def __init__(self, proxy_credentials):
        self.proxy_credentials = proxy_credentials
        self.isValid = None
        self.attempts = 0

    def test(self):
        global lock_

        try:

            r = requests.get("http://google.com/", proxies={"http": f"http://{self.proxy_credentials}"},
                             timeout=self.timeout)
            self.isValid = True

        except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
            self.isValid = False

        with lock_:
            self.attempts += 1


# Making a proxy queue, you can ignore this, just remember that the queue name is queue_

queue_ = queue.SimpleQueue()
string_proxies = []

for x in range(0, 20):
    string_proxies.extend(["120.237.144.77:9091", "139.9.62.22:80", "120.237.144.77:9091"])

for proxy in string_proxies:
    queue_.put(Proxy(proxy))

# Thread testing of queue
max_attempts_per_proxy = 1


def print_test_proxy(proxy_object):
    global lock_
    global queue_

    proxy_object.test()

    with lock_:
        if proxy_object.isValid:
            print(f"Working - {proxy_object.proxy_credentials}")
        else:
            print(f"Bad - {proxy_object.proxy_credentials}")

    if proxy_object.attempts < max_attempts_per_proxy:
        queue_.put(proxy_object)


with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    timeout = Proxy.timeout + 2
    while True:
        try:
            proxy = queue_.get(timeout=timeout)
            executor.submit(print_test_proxy, proxy)
        except (queue.Empty, KeyboardInterrupt):
            with lock_:
                print("No more threads are being created.")

In this code sample I am unable to catch KeyboardInterrupt correctly before the "No more threads are being created" statement is shown, I will get this exception:

    Traceback (most recent call last):
      File "C:***\testing2.py", line 69, in <module>
        proxy = queue_.get(timeout=timeout)
    _queue.Empty
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "C:\Users\coyot\Desktop\Checker\testing2.py", line 69, in <module>
        proxy = queue_.get(timeout=timeout)
    KeyboardInterrupt

My hypothesis on why KeyboardInterrupt isn't being catched properly, is that the queue.Empty exception is raised, however, because of multiple threads, the print message is not shown instantly. That's why you may think that threads aren't in the process of stop being created, so you hit CTRL + C, but the main thread is already on the catch block, and there is not a try catch block for the KeyboardInterrupt in there.

If you set the global variable of max_attempts to a higher number though, you will be able to catch the exception correctly, hence why I think this.

However, in the 1 max_attempt scenario, if we keep in mind this, and try to catch KeyboardInterrupt within the block that's catching the queue.Empty exception, I am unable to catch it either way.

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    timeout = Proxy.timeout + 2
    while True:
        try:
            proxy = queue_.get(timeout=timeout)
            executor.submit(print_test_proxy, proxy)
        except (queue.Empty, KeyboardInterrupt):
            try:
                break
            except KeyboardInterrupt:
                break

    with lock_:
        print("No more threads are being created.")

I get this exception:

Traceback (most recent call last):
  File "C:***\testing2.py", line 69, in <module>
    proxy = queue_.get(timeout=timeout)
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:***\testing2.py", line 69, in <module>
    proxy = queue_.get(timeout=timeout)
KeyboardInterrupt

I would like to be able to catch both the queue.Empty exception and the KeyboardInterrupt too, I'd like an explanation of why i'm not able to do it with the method I provided.

Edit: @Michael Butscher idea works as expected.

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
    timeout = Proxy.timeout + 2
    while True:
        try:
            try:
                proxy = queue_.get(timeout=timeout)
                executor.submit(print_test_proxy, proxy)
            except queue.Empty:
                with lock_:
                    print("No more threads are being created due to an empty queue")
                break
        except KeyboardInterrupt:
            with lock_:
                print("Stopped while loop flow of CTRL + C")
            break

    with lock_:
        print("Work of threads has ended")

Although this works, it seems odd that my solution didn't work. As a similar worflow works in a non-multithreaded scenario.

try:
    print("First try")
    raise Exception

except Exception:
    try:
        print("Try inside exception")
        raise Exception
    
    except Exception:
        print("Program worked as expected")
0 Answers
Related