I need help building a bot with python selenium chrome driver that bypasses gumtree.com.au login page. When the the chrome driver goes to the specified page
It loads nothing, just a blank white page. However, when I plug it into any browser manually, it loads perfectly. I can't login get it to load with selenium chrome driver at all and I've tried changing headers, using headless driver, and using vim to change all cdc_ texts globally (yes, its correct, I've triple checked).
Used this useragent class (actually have to import package fake-useragent):
from fake_useragent import UserAgent
Snippet of headless driver:
options.add_argument(f'user-agent={random_header}')
options.add_argument("--window-size=1920,1080")
options.add_argument('--ignore-certificate-errors')
options.add_argument('--allow-running-insecure-content')
options.add_argument("--disable-extensions")
options.add_argument("--proxy-server='direct://'")
options.add_argument("--proxy-bypass-list=*")
options.add_argument("--start-maximized")
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')
options.add_argument('--no-sandbox')
options.add_argument(f"user-agent={random_header}")
Couple of questions:
- What should I do? Open to using other automated tool as well.
- How to build a bot that goes undetected to most sites, if not all? Especially the ones that are trying to prevent scraping (I'm not scraping, I'm just assuming they get it the worst)