How to automate a headless, undetected chrome browser with proxy on Ubuntu Server using python selenium

Viewed 265

I am automating a task on python selenium to visit a website where I am unable to get access. This browser or app may not be secure Try using a different Browser. As a solution, I am using an undetected chrome browser with a proxy.

When automation starts we get a proxy alert for authentication "sign in the proxy requires a username and password. Your connection to this site is not private" enter image description here

To handle this I have used the library PyAutoGui. This fills the user name and password for the proxy and the whole automation is successful.

The problem is that I want to run this on the ubuntu server for which I have to use headless but the pyautogui does not work on headless chrome. It detects an open window on the screen and throws the username and password there (in the code script of PYCHARM in my case). So because the proxy is not authenticated as it did not get the required values, the page does not load and the selenium automation throws an error due to the element it is expecting on the page to appear.

File "C:\Users\username\PycharmProjects\ProjectName\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 90, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 
Stacktrace:

I have tried these solutions but could not find where all three are working:

  • Undetected browser
  • with proxy
  • running headless

I have tried so many things mostly shared in undetected-chromedriver issues but when it comes to headless all these solutions fail

using extensions with zip

Using selenium wire proxy options

The selenium does not seem to access with selenium alert!

Here is my Code (pseudo in some parts)

import pandas as pd
from imap_tools import MailBox, AND
import re
# No Headless, Proxy, undetected
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
from threading import Thread
import pyautogui
from selenium.webdriver.chrome.options import Options
import chromedriver_autoinstaller
import undetected_chromedriver as uc
from seleniumwire import webdriver
from seleniumwire import webdriver

# host_ip = "xx.xxx.xx.xx"
host_ip = "x.xxx.xxx.xxx"
port = "xxxxx"
proxy_username = "xxxxxxxx"
proxy_password = "xxxxxxxx"
ipCheckURL="http://whatismyip.com"

URL=undetectedCheckUrl


def enter_proxy_auth(proxy_username, proxy_password):
    time.sleep(3)
    pyautogui.typewrite(proxy_username)
    pyautogui.press('tab')
    time.sleep(5)
    pyautogui.typewrite(proxy_password)
    pyautogui.press('enter')
    time.sleep(15)

def visitWebsiteAndAutomateAndReturnAValue(driver, url):

    driver.get(URL)
    #Do some Automation
    return Value



    chrome_options = Options()
    chrome_options.add_argument('--proxy-server={}'.format(host_ip + ":" + port))
    ucdriver = uc.Chrome(options=chrome_options, use_subprocess=True)
    x1 = Thread(target=visitWebsiteAndAutomateAndReturnAValue, args=(ucdriver, url))
    x2 = Thread(target=enter_proxy_auth, args=(proxy_username, proxy_password))
    x1.start()
    x2.start()
    value = x1.join()
    x2.join()
    print(value)

So far In similar posts, I do not see a solution but people just posting their Wikipedia knowledge on cybersecurity or how the internet works. Please if you know how I can handle this do share, I will really appreciate it.

1 Answers

Try using seleniumwire, it supports proxy with authentication.

from seleniumwire import webdriver
options = {
    'proxy': {
        'https': 'https://user:pass@192.168.10.100:8888',
    }
}
driver = webdriver.Chrome(seleniumwire_options=options)
Related