Chrome Headless in AWS Lambda returns empty page

user_8070551 0
Viewed 237

Am using Chrome Headless (with Serverless framework) to run my selenium scraping script in an AWS Lambda function.

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from time import sleep

def main(event, context):
    options = Options()
    options.binary_location = '/opt/headless-chromium'
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument('--single-process')
    options.add_argument('--disable-dev-shm-usage')
    driver = webdriver.Chrome('/opt/chromedriver',chrome_options=options)

    driver.get('https://www.linkedin.com/in/williamhgates')
    sleep(2)
    body = f"Headless Chrome Initialized, Page : {driver.page_source}"

    driver.close()
    driver.quit()

    response = {
        "statusCode": 200,
        "body": body
    }

    return response

Same script works perfectly in my local linux machine, returning the good source page. But when I'm using it though AWS Lambda, it's returning an empty page with the following source code :

<html xmlns=\"http://www.w3.org/1999/xhtml\"><head></head><body></body></html>

Do you have any ideas ? Thank you in advance

aws-lambda web-scraping selenium chromium
1 Answers

This seems to be an issue with the SSL certificate, set the desired capabilities to ignore it

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities


desired_capabilities = DesiredCapabilities.CHROME.copy()
desired_capabilities['acceptInsecureCerts'] = True
user_142445880
Related