Using Selenium with Chrome driver on Ubuntu server which does not have GUI

I want so scrape some dynamic url (the page is built using JavaScript and does a redirect to an external page). I understand that I need to use a headless browser and I am using Selenium with Chrome driver. The following code does what I want on my Windows machine:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        self.driver = webdriver.Chrome(ChromeDriverManager().install())

    def parse(self, response):
        link = "some-url-which-uses-javascript.com"
        self.driver.get(link)
        time.sleep(1) # without this wait, driver.current_url is not the final redirect
        url = self.driver.current_url

But when I run the same code on my Ubuntu server (which does not have GUI) I get the following error:

builtins.ValueError: Could not get version for Chrome with this command: google-chrome --version || google-chrome-stable --version

I have installed both google-chrome-stable and chromium-chromedriver on the Ubuntu server.

I have also tried the following code:

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        # self.driver = webdriver.Chrome(ChromeDriverManager().install())
        # self.driver = webdriver.Chrome('/usr/bin/chromedriver')
        chrome_options = Options()
        chrome_options.add_argument('--headless')
        chrome_options.add_argument('--no-sandbox')
        chrome_options.add_argument('--disable-dev-shm-usage')
        self.driver = webdriver.Chrome('/usr/bin/chromedriver', chrome_options=chrome_options)

Bu I get the following error:

selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally. (unknown error: DevToolsActivePort file doesn't exist) (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)