Why is guild.members.fetch timing out

Hello so I am writing a discord bot and all I want is an ordered list of members ordered by their ids. To achieve this I am running following code, however it just console.logs "Couldn't fetch members", with no further errors. :(

Upon further inspection, I discovered that it is a timeout error. Everything else works fine and I think that the const guild isn't the problem, but other than that hope that you people can help, Cheers!

const IDs = new Map();
var repeat = new Boolean(false);
var randomInt = new Number(0);

client.on('ready', () => {
    console.log('Ready!');
    const guild = client.guilds.cache.get("xxxxxxxxxxxxxxxxxx");
    guild.members.fetch().then(members => {
        console.log("Found the members");
        IDs = members.map(user => user.id);
        const OnlineMembers = members.filter(member => member.presence.status == "online");
    }).catch(e => console.log("Couldn't fetch members."));
});

P.S. This is running on a Raspberry Pi over Nodemon --inspect, in case that changes anything