I need to run multiple (i.e. 100) scans parallelly using selenium but because of the high CPU usage, my server is getting crushed very quickly. I'd like to know how can I significantly decrease high CPU usage because even if I run 5 scans at a time the CPU usage still hits more than 50%? Should I use another webdriver or can I somehow slow down the scan time but save CPU power(i.e. apply limits)? I've applied some options to decrease the CPU utilization but it didn't help. Below is the chrome options
def get_chrome_options():
options = webdriver.ChromeOptions()
options.add_argument('headless')
options.add_argument('window-size=1200x600')
options.page_load_strategy = 'normal'
options.add_argument('--no-sandbox')
options.add_argument("--disable-setuid-sandbox")
options.add_argument("--disable-dev-shm-usage")
options.add_argument('--disable-gpu')
options.add_argument("--disable-extensions")
options.add_argument('--ignore-certificate-errors')
options.add_argument("--FontRenderHinting[none]")
return options
I need to create and run a web driver from my web app (Django) so I have to create a new webdriver each time I make an HTTP request and I can't reuse the same webdriver. Below is the example.
def run_scan(request, url):
driver = webdriver.Chrome(executable_path=settings.CHROME_DRIVER, options=get_chrome_options())
driver.get(url)
print(driver.page_source)
Below info describes my current environment
Python - 3.9.7
Selenium - 3.141.0
Chromedriver - 93.0.4577.82
OS - alpine linux