How to rethink the architecture of my Python project containing multiple async interfaces

Viewed 84

I'm developping a Twitch Bot for approximatively a year now. Over the time, the bot became bigger and bigger to add features. Now the bot can manage multiple interfaces including Discord, Twitch, Twitch API, Streamlabs... and have a web interface to receive all the callbacks from OAuth authentification APIs and to present some statistics on an HTML page to the streamer.

However, the bigger the bot gets, the more problems I experience. The fact is, I don't think my current architecture is good. This is how it's done currently :

First, I instantiate a Core class, this class will contain instance of all the bots and interface and store the variables shared between them. It also contains a global asyncio lock for the database (since the entire bot is running on a single async loop, I need to be sure only one interface is talking with the database at a same time).

class Core:
    def __init__(self):
        # Define session lock for database access
        self.session_lock: asyncio.locks.Lock = asyncio.Lock()

        # Store bots & API access
        self.discord_bot: Optional[discord.Client] = None
        self.twitch_bot: Optional[twitchio.ext.commands.Bot] = None  # Manage Twitch IRC chat
        self.twitch_api: TwitchApi = None  # Manage Twitch API
        # Other interfaces ...

        # Some shared attributes which are read by all interfaces ...

Then, I instantiate all my interfaces by passing them the core. Each interface will register into the core by itself when instantiated. This is an example of an interface initialization (Discord here) :

class DiscordBot(commands.Bot):
    def __init__(self, core: Core, **options):
        super().__init__(**options)

        self.core: Core = core
        self.core.discord_bot = self

And the instanciation phase in my main script:

core = Core()

discord_bot = DiscordBot(core)
twitch_bot = TwitchChatBot(core, os.environ['TWITCH_BOT_TMI_TOKEN'], [os.environ['TWITCH_CHANNEL_NAME']])

loop = asyncio.get_event_loop()
loop.create_task(twitch_bot.connect())
loop.create_task(discord_bot.start(os.environ["DISCORD_BOT_TOKEN"]))
loop.run_forever()

Here is a global diagram of my architecture and how it is managed :

enter image description here

This architecture is really convenient because it allows me to have a really simple bridge between interfaces. For example, if I want to post a message on Discord from my Twitch bot, I can simply call self.core.discord_bot.get_channel(...).send(). Same thing on the other direction.

But I feel like this architecture is no longer sustainable. Currently the Core class contains more than 5000 lines of code and more than 60 methods shared between all the interfaces. I would like to explode it in multiple file but it's a mess. Moreover, I increasingly think that running all interfaces on the same asynchronous loop is not a good idea in the long run.

I thought of solutions, like separating all the interfaces into different processes. But then how to manage the synchronization between these processes without doing something complicated (I take my example of posting a message on Discord from the Twitch bot). I also looked at solutions like Redis for the synchronisation but I don't really know if it could answer all my concerns... I also thought about using the python import module to directly import a Core instance (and thus not having to register the Core in each interface), but I'm not sure if it would work and if it's a good practice.

I take another example, in the Core class is instanciated a class processing the interactions between the bot and the community. It is used to vary the interactions with the users (a sort of primitive chatbot). This class MUST be shared between all my interfaces because I want the reactions of the Discord bot to react according to what is happening on Twitch.

Anyway, I would like your expert opinion on this. How would you have organized all this? Thanks :)

1 Answers

There is something unclear from your description, and I think it might be important:

Are the bots at the bottom of the diagram connecting the the foreign services and then providing an API interface to the core, or are they driving the core to do the work?

In Enterprise speak: Where is the business logic?

I think I have been building a similar system.

In mine:

  • There a small set of services: (I am not sure if these are analogous to your bots.)

    • An interface to Twitch chat.
    • An interface to Twitch redemptions.
    • An interface to KoFi (so I am can react to donations in real time)
    • Arguably the logging system could be considered one.
    • I don't have a database, but if I did, the service would go here.
    • I don't use YouTube, but if I did, the service would go here.
    • I've considered putting my StreamLabs Desktop API interface here [via PySLOBS] but I haven't bothered yet.
    • I've considered putting an common asynchio loop here, but I haven't bothered yet.
  • There is the framework:

    • It instantiates and owns the survives.
    • It registers, starts and stops the plugins (below).
  • There are a larger set of plugins.

    • Each plugin only does one thing.
      • e.g. if someone redeems the TextToSpeech reward, it plays the speech. The plugin subscribes to redemptions via the service, and acts when triggered.
      • e.g. if someone types "GG" in chat, it plays an animation on stream. The plugin subscribes to chat via the service, and connects to StreamLabs when triggered.
    • Some plugins are pretty passive; they act when called back by the main app.
    • Some plugins have separate threads, so they can act independently.
    • Some plugins have threads that spawn asyncio loops (especially because PySLOBS using asyncio).
    • Some plugins have separate processes (especially so that Kivy or PyGame windows can be spawned by them).

It sounds to me like you have put all of the code that I put into separate, independent, loosely-coupled plugins into one place in the Core, and it has become unwieldy.

Consider breaking that out into individual parts that have the same interface (start, stop and maybe some status for monitoring) but take care of only one piece of functionality. Most are probably only going to talk to one or two services, and be pretty simple.

Related