I'm using Selenium with chrome driver to scrape facebook, insta and reddit. Everything's working perfectly on local server. However, when I deploy on AWS EC2 (using Elastic Beanstalk) it does not work properly. Reddit is being scraped without any problem but for facebook and instagram it throws: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element.
Searching for the solution to this problem, I came to know that this error occurs when the page is not yet loaded completely and we try to look for elements of the page. (Note: this issue did not occur on my local machine.) To resolve this problem I added waiting delay so that the page can load completely. I tried giving delay of up to 60 seconds but it is still failing to load the page. Note: this issue does not occur on page 1 e.g. on facebook: login goes successful, search query goes successful and the results are shown and then when a try is made to load a specific post error occurs (page fails to load) similarly on Instagram: login is successful but when a try is made to access the search box element error occurs.
No idea why the pages are not being loaded when accessing through AWS server but working fine on my local server. The fact that reddit is being scraped confirms that the issue is not with selenium or other environmental dependencies...
If someone has any idea about this problem, please share. Thanks.