"Chrome not reachable" using Xvfb in non headless mode

I am getting the following error:

  File "/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 333, in get
    self.execute(Command.GET, {'url': url})
  File "/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: chrome not reachable
  (Session info: chrome=80.0.3987.122)

when I try to run the Selenium Chromedriver without --headless option and using xvfb.

It works perfectly in headless mode without the xvfb. I need xvfb because I need it not in headless mode in order to run browser extensions, which are not supported otherwise.

Why is this happening? Is there any way to achieve what I want at all?