To quickly explain how this code works, we have an object called Proxy, we have a queue with a lot of these objects, and they are being multithreaded to check if you can succesfuly do a request with its credentials.
The problem comes in the manner that, I get an item from the queue in the main thread, having a timeout on the get() queue method (which value is a couple seconds more than the timeout that we have set on the Proxy object class att for the request). This happens in a while True with a try catch which is catching both the timeout exception and KeyboardInterrupt, however, I am unable to catch KeyboardInterrupt in some cases, which I will explain below
import concurrent.futures
import queue
import threading
import requests
lock_ = threading.Lock()
class Proxy:
timeout = 3
def __init__(self, proxy_credentials):
self.proxy_credentials = proxy_credentials
self.isValid = None
self.attempts = 0
def test(self):
global lock_
try:
r = requests.get("http://google.com/", proxies={"http": f"http://{self.proxy_credentials}"},
timeout=self.timeout)
self.isValid = True
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout):
self.isValid = False
with lock_:
self.attempts += 1
# Making a proxy queue, you can ignore this, just remember that the queue name is queue_
queue_ = queue.SimpleQueue()
string_proxies = []
for x in range(0, 20):
string_proxies.extend(["120.237.144.77:9091", "139.9.62.22:80", "120.237.144.77:9091"])
for proxy in string_proxies:
queue_.put(Proxy(proxy))
# Thread testing of queue
max_attempts_per_proxy = 1
def print_test_proxy(proxy_object):
global lock_
global queue_
proxy_object.test()
with lock_:
if proxy_object.isValid:
print(f"Working - {proxy_object.proxy_credentials}")
else:
print(f"Bad - {proxy_object.proxy_credentials}")
if proxy_object.attempts < max_attempts_per_proxy:
queue_.put(proxy_object)
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
timeout = Proxy.timeout + 2
while True:
try:
proxy = queue_.get(timeout=timeout)
executor.submit(print_test_proxy, proxy)
except (queue.Empty, KeyboardInterrupt):
with lock_:
print("No more threads are being created.")
In this code sample I am unable to catch KeyboardInterrupt correctly before the "No more threads are being created" statement is shown, I will get this exception:
Traceback (most recent call last):
File "C:***\testing2.py", line 69, in <module>
proxy = queue_.get(timeout=timeout)
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\coyot\Desktop\Checker\testing2.py", line 69, in <module>
proxy = queue_.get(timeout=timeout)
KeyboardInterrupt
My hypothesis on why KeyboardInterrupt isn't being catched properly, is that the queue.Empty exception is raised, however, because of multiple threads, the print message is not shown instantly. That's why you may think that threads aren't in the process of stop being created, so you hit CTRL + C, but the main thread is already on the catch block, and there is not a try catch block for the KeyboardInterrupt in there.
If you set the global variable of max_attempts to a higher number though, you will be able to catch the exception correctly, hence why I think this.
However, in the 1 max_attempt scenario, if we keep in mind this, and try to catch KeyboardInterrupt within the block that's catching the queue.Empty exception, I am unable to catch it either way.
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
timeout = Proxy.timeout + 2
while True:
try:
proxy = queue_.get(timeout=timeout)
executor.submit(print_test_proxy, proxy)
except (queue.Empty, KeyboardInterrupt):
try:
break
except KeyboardInterrupt:
break
with lock_:
print("No more threads are being created.")
I get this exception:
Traceback (most recent call last):
File "C:***\testing2.py", line 69, in <module>
proxy = queue_.get(timeout=timeout)
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:***\testing2.py", line 69, in <module>
proxy = queue_.get(timeout=timeout)
KeyboardInterrupt
I would like to be able to catch both the queue.Empty exception and the KeyboardInterrupt too, I'd like an explanation of why i'm not able to do it with the method I provided.
Edit: @Michael Butscher idea works as expected.
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
timeout = Proxy.timeout + 2
while True:
try:
try:
proxy = queue_.get(timeout=timeout)
executor.submit(print_test_proxy, proxy)
except queue.Empty:
with lock_:
print("No more threads are being created due to an empty queue")
break
except KeyboardInterrupt:
with lock_:
print("Stopped while loop flow of CTRL + C")
break
with lock_:
print("Work of threads has ended")
Although this works, it seems odd that my solution didn't work. As a similar worflow works in a non-multithreaded scenario.
try:
print("First try")
raise Exception
except Exception:
try:
print("Try inside exception")
raise Exception
except Exception:
print("Program worked as expected")