DiscordJS - asynchronously respond to messages

user_7124232 0
Viewed 315

One of the commands in a discord bot I'm writing involves scraping a website using JSDom, waiting for its scripts to all run before reading the HTML. What I've found is that sometimes it takes a long time for that site's scripts to run, which causes the bot to become unresponsive for 3 minutes while it waits for the last query to finish. How can I make the action of scraping into an asynchronous function, so that the bot can still respond to other messages while it waits for the first call to finish?

This is a much-simplified version of my current code:

client.on('message', message => {
    // command1 - the web scraper
    console.log("benchmark A");
    jsdom.JSDOM.fromURL(link, {resources: "usable", runScripts: "dangerously"}).then(dom => {
        console.log("benchmark B");
        dom.window.document.addEventListener('DOMContentLoaded', () => {
            console.log("benchmark C");
            setImmediate(() => {
                console.log("benchmark D");
                // read the DOM and do stuff with it
                message.channel.send(some_value_from_dom);
            });
        });

    });
    // command2
    // command3
    // command4
    // etc.
});

You can see I included some log statements. When someone calls command1, what happens in the console is that benchmarks A, B, C all show up almost immediately, but then benchmark D takes up to 3 minutes to appear. The main problem that I'm having is that, in that time between benchmark C and benchmark D, nobody else can query my bot. So I want to move the webscraping portion of the code into its own asynchronous function somehow, so that it can run in the background and not hold up the main block, thereby letting others continue to use the bot while this time-consuming command is processed. But I have no clue how to go about doing this in NodeJS since JavaScript doesn't support multithreading.

discord.js node.js asynchronous javascript
1 Answers

A little bit more digging and I've solved my own problem. I placed the webscraping function into a worker thread, and each time I want to it to run I post the link from the main to the worker, let it do its thing, then when it's done it posts the message back to the main thread.

Here's what the code looked like before:

// bot.js

client.on('message', message => {
    jsdom.JSDOM.fromURL(link, {resources: "usable", runScripts: "dangerously"}).then(dom => {
        dom.window.document.addEventListener('DOMContentLoaded', () => {
            setImmediate(() => {
                // read the DOM and do stuff with it
                message.channel.send(some_value_from_dom);
            });
        });

    });
});

And here's what it looks like after:

// bot.js

const worker = new Worker('./worker.js');

client.on('message', message => {
    // Construct link
    worker.once('message', response => {
        message.channel.send(response);
    });
    worker.postMessage(link);
});

// worker.js

parentPort.on('message', link => {
    jsdom.JSDOM.fromURL(link, {resources: "usable", runScripts: "dangerously"}).then(dom => {
        dom.window.document.addEventListener('DOMContentLoaded', () => {
            setImmediate(() => {
                // read the DOM and do stuff with it
                parentPort.post(some_value_from_dom);
            });
        });
    });
});

Now all the webscraping is done separately and other users can still query the bot's other commands in the meantime, without having to wait several minutes for the first task to be done. Maybe there's a simpler solution that I missed, but a) I won't claim to be a JS expert and b) this works exactly the way I want it to, so I see no reason to change it again.

user_71242320
Related