I'm trying to reduce the amount of time that requests I make using pycurl library uses. I've noticed that reusing handles instead of creating them during every connection results in a significant reduction in overhead.
So, I've started to use pre-created curl handles, that only require for the pycurl.URL attribute to be set for basic "GET" requests, and are reusable (along other optimisations). However, when threading it, I would rather create a copy of the handle to use, but no copying methods I work try.
I tried to do a deepcopy() of the pre-created curl handles, and I received an error saying that "TypeError: Curl objects do not support serialization". Is there any other way to bypass this, as I'd rather avoid the overhead produced from creating one handle per thread.
Update: I've come up with a solution to reduce overhead as much as possible, without using copies. However, my plan is to test how good is the performance compared with doing curlHandle copies