Concurrency questions with non-concurrent code

alb2001 0
Viewed 23

I have a library that does calls to smart contracts in the ethereum chain to read data

So for simplicity, my code is like this:

import library

items = [
"address1",
"address2",
"address3",
]

for item in items:
    data = library.get_smartcontractinfo(item)
    print(data)

if __name__ == '__main__':
    main()

I am new to concurrency and this is a topic I need to explore further, as there are many options to do concurrency but seems asyncio is the one most people go for

The library I a musing is not built with asyncio or any sort of concurrency in mind. This means that each time I call the library.get_smartcontractinfo() function then I need to wait until it completes the query so it can do the next iteration, which is blocking the speed.

Lets say that I cannot modify the library, althought maybe I will in the future, but I wanto get something done asap with the existing code

What would be the easiest way to do simultaneous queries so I can get the info as fast as I can in an efficient way?

What about being rate limited? And would it be possible to group these calls into one without rewriting the library code?

Thank you.

ethereum python
2 Answers

Assuming that library.get_smartcontractinfo() does a lot of network I/O, you could use a ThreadPoolExecutor from concurrent.futures to run more of them in parallel.

The documentation has a good example.

userFd750

Assuming the function library.get_smartcontractinfo() is a I/O bound, you have multiple options to go with asyncio. If you want to use pure asyncio you can go with something like

async def main():
    loop = asyncio.get_running_loop()
    all_runs = [loop.run_in_executor(None, library.get_smartcontractinfo, item) for item in items]
     results = await asyncio.gather(*all_runs)

Bascially running the sync function in a thread. To run those concurrently, you first create all coroutines without awaiting them, and finally pass those into gather.

If you want to use some additional library, I can recommend using anyio or asyncer which basically is a nice wrapper around anyio. With `asyncer?, you basically can change the one line where you transfer a sync function into an async one to

from asyncer import asyncify
...
all_runs = [asyncify(library.get_smartcontractinfo)(item) for item in items]

the rest stays the same.

Simon Hawe0
Related