Chrome Driver Error using Selenium on AWS: unable to discover open window in chrome

I want to run Selenium in AWS, specifically in a Lambda function. The Lambda function should use a Docker Image, which is stored on Amazon ECR. The Selenium code is simple Demo code in Python.

When I run a Docker Container from my image locally, everything works fine and Selenium succeeds, but when I deploy the image to AWS and configure the lambda function to use it, it fails and throws this error:

selenium.common.exceptions.WebDriverException: Message: unknown error: unable to discover open window in chrome

I have already tried all possible chrome driver arguments and even tried the solutions from this question: Aws Lambda Ruby Crawler selenium chrome driver: 'unknown error: unable to discover open window in chrome'

Nothing seems to work. The error doesn't disappear.

I have used different browsers including google-chrome-stable, firefox, headless-chromium with different versions, all fail in aws, but work in a local docker container.

I also tried to deploy a Debian image with the required dependencies instead of aws-lambda-python, but it also throws the same error.

Currently I am using Google Chrome Version 91, chromedriver 91.0.4472.101 for linux64 and selenium 3.141.0. The Dockerfile looks like this:

FROM amazon/aws-lambda-python:3.8
RUN curl https://dl.google.com/linux/direct/google-chrome-stable_current_x86_64.rpm > google-chrome-stable_current_x86_64.rpm
RUN yum install ./google-chrome-stable_current_x86_64.rpm -y
RUN pip install selenium
COPY . ./
CMD [ "main.py" ]
ENTRYPOINT [ "python" ]

main.py:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys


if __name__ == '__main__':
    chrome_options = webdriver.ChromeOptions()
    chrome_options.add_argument("--headless")
    chrome_options.add_argument("--disable-gpu")
    chrome_options.add_argument("--disable-application-cache")
    chrome_options.add_argument("--disable-infobars")
    chrome_options.add_argument("--no-sandbox")
    chrome_options.add_argument('--disable-dev-shm-usage')
    chrome_options.add_argument("--hide-scrollbars")
    chrome_options.add_argument("--enable-logging")
    chrome_options.add_argument("--single-process")
    chrome_options.add_argument("--ignore-certificate-errors")
    chrome_options.add_argument("--homedir=/tmp")
    chrome_options.add_argument("--log-level=0")
    driver = webdriver.Chrome(executable_path="/var/task/chromedriver", options=chrome_options)
    driver.get("http://www.python.org")
    assert "Python" in driver.title
    elem = driver.find_element_by_name("q")
    elem.clear()
    elem.send_keys("pycon")
    elem.send_keys(Keys.RETURN)
    assert "No results found." not in driver.page_source
    driver.close()
    print("done")

Directory Structure

EDIT 1: I have tried using the selenium/standalone-chrome and selenium/standalone-firefox image like Moshe Slavin suggested. On chrome, I am now getting a different error with the same options (pages instead of window):

selenium.common.exceptions.WebDriverException: Message: unknown error: unable to discover open pages

On firefox I get this error. Exactly the same error before using the selenium image:

selenium.common.exceptions.WebDriverException: Message: Process unexpectedly closed with status signal