convert async object into a list

Question

I'm trying to get a list of proxies from proxybroker:

import asyncio
from proxybroker import Broker

async def show(proxies):
    while True:
        proxy = await proxies.get()
        if proxy is None: break
        print('Found proxy: %s' % proxy)

proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
    broker.find(types=['HTTP', 'HTTPS'], limit=10),
    show(proxies))

loop = asyncio.get_event_loop()
result_now = loop.run_until_complete(tasks)

However, I'm not sure how to convert the final result into a normal list. Any help? Thanks!

user_5129881 · Answer

It seems the code in your question is copy-pasted from Python: Correct way of using proxybroker lib with asyncio

Check it out again to see what this snippet is actually supposed to do (print proxies, instead of returning them)

As per your question, to fill result_now with the proxies generated by the broker you will need to return a value from the "reader" coroutines, which consumes async events produced through the queue by the broker.

An example with a mock replacing broker:

import asyncio

async def broker_mock(proxy_queue):
    for item in [1, 2, 3, 4, 5]:
        proxy = await proxy_queue.put(item)
    await proxy_queue.put(None)

async def show(proxy_queue):
    found = []
    while True:
        proxy = await proxy_queue.get()
        if proxy is None: break
        found.append(proxy)
    return found

proxy_queue = asyncio.Queue()
tasks = asyncio.gather(broker_mock(proxy_queue), show(proxy_queue))
loop = asyncio.get_event_loop()
result_now = loop.run_until_complete(tasks)
print(result_now[1])
# should printout: [1, 2, 3, 4, 5]

I recommend reading https://docs.python.org/3/library/asyncio-task.html for further explanation on asyncio.

user_6083423 · Answer

Your code is the basic example from ProxyBroker's GitHub repository. I looked at the Broker function's source code to see if there was some parameter that could be enabled. The only parameter was writing the proxy query results to a file, which isn't what you requested.

I noted that when I requested type(proxy) it returned object. These objects can be appended to a list in your show(proxies) function, which you can return after loop.run_until_complete(tasks).

There are others ways to accomplish your use case, but the one below is quick and accomplishes your requirement with minimal code changes.

import asyncio
from proxybroker import Broker

proxy_data = []

async def show(proxies):
    global proxy_data
    while True:
        proxy = await proxies.get()
        if proxy is None:
            break
        # the `str` allows you to manipulate the list items
        proxy_data.append(str(proxy))


proxies = asyncio.Queue()
broker = Broker(proxies)
tasks = asyncio.gather(
    broker.find(types=['HTTP', 'HTTPS'], limit=10), show(proxies)
)

loop = asyncio.get_event_loop()
loop.run_until_complete(tasks)


for item in proxy_data:
    print(item)
    # output

convert async object into a list

2 Answers