Can I make the python code work while sorting list

user_10901179 0
Viewed 139

I did a top command on discord.py (using Replit database). Basically shows the top 10 users with the most XP in the server.

That's how a part of my code looks like, ignore my messy way of saving keys to the database

@commands.command()
async def top(self,ctx,nr:int=1):
      match_keys = db.prefix(f"{ctx.guild.id}XP")
      sort = sorted(match_keys, key=lambda x: db[x],reverse=True)
[...]

I have like 2000 key to sort, this whole thing takes about 10 seconds to finish. In those 10 seconds, the bot is unusable. I have tested out and the only thing that takes time is the sort. I saw other similar questions on StackOverflow but from what I found there, the sort/sorted method is based on quicksort, I can't really do much about it to be faster.

I am thinking of making a task loop which will update the top once in a while (less time to wait when running the .top command), but the bot will be unusable while it sorts the list.

Is it possible to slow down the sort so I could use the bot while the operation? Thanks!

sorry for potential mistakes, I'm new to stackoverflow

discord.py sorting python
2 Answers

As I learned from the comments, your database doesn't support multi-key/batch queries. However, there are still some approaches you can take, like caching, awaitable remote procedures, dumping the whole database, etc.

A quick solution could be that you separate querying the database from sorting the entries giving you the possibility to use an async function - and not to block the whole ten seconds.

async def get_entries(match_keys):
    entries = {}
    for key in match_keys:
        entries[key] = db[key]
        await asyncio.sleep(0) # yield control to the event loop
    return entries

@commands.command()
async def top(self, ctx, nr:int=1):
      match_keys = db.prefix(f"{ctx.guild.id}XP")
      entries = await get_entries(match_keys)
      sort = sorted(entries.items(), key=lambda e: e[1], reverse=True)
user_26124840

To avoid blocking the event loop (and thus an async application) with long-running, synchronous tasks it is straightforward to offload them to a thread. In fact, asyncio already has a builtin thread pool which can run such tasks.

Since Python3.9, asyncio.to_thread makes this convenient:

import asyncio

@commands.command()
async def top(self,ctx,nr:int=1):
      match_keys = db.prefix(f"{ctx.guild.id}XP")
      sort = await asyncio.to_thread(
          sorted, match_keys, key=lambda x: db[x], reverse=True
      )

Earlier versions can use asyncio's loop.run_in_executor instead:

import asyncio
from functools import partial

@commands.command()
async def top(self,ctx,nr:int=1):
      match_keys = db.prefix(f"{ctx.guild.id}XP")
      loop = asyncio.get_running_loop()
      sort = await loop.run_in_executor(
          None,
          partial(sorted, match_keys, key=lambda x: db[x], reverse=True),
      )
user_53499160
Related