Web scraper works locally but not when deployed to Heroku server

user_15506187 0
Viewed 702

So I'm trying to use the SecretAgent web scraper and it works when I run node app.js but when I deploy my file structure to Heroku, it doesn't work and I get the following massive error:

https://i.stack.imgur.com/YNygo.jpg

https://i.stack.imgur.com/KwVr4.jpg

https://i.stack.imgur.com/iSKKv.jpg

My call w/ SecretAgent looks like this:

const agent = require('secret-agent');

(async () => {
  await agent.goto('https://example.org');
  const title = await agent.document.title;
  const intro = await agent.document.querySelector('p').textContent;
  console.log("intro is " + intro);
  await agent.close();
})();

I also tried entering the follow sudo command after cd'ing into the server folder but ended up getting the error 'UnhandledErrorOrRejection', "clientError: 'TypeError: coreTab.goto is not a function'":

sudo $(npx puppet-install-deps)

This is where I currently am. Any help would be much appreciated. If I can clarify the question better, let me know--thank you for all of your help!

buildpack node.js web-scraping heroku sudo
1 Answers

I've never used Secret Agent and I certainly haven't deployed a web scraper to Heroku, but it looks like in the Docs Secret Agent tries to store your session data automatically. As far as I understand it, Heroku offers no native storage. Instead, you can hook it up to a database. I believe what is happening is that your Secret Agent isn't able to store session information, and crashes. Is there a reason you can't run the scraper locally? If you must do it on Heroku, you will need to connect a database (like JawsDB) and use the sessionDir to force it to store your information there.

user_140898900
Related