New to Stack Overflow posting so apologies in advance.
I am trying to use selenium + headless chrome to perform browser automation, but in initial testing noticed some odd behavior with saving cookies and other local data when comparing headless vs non headless chrome instances. For reference, my chrome version is: Version 87.0.4280.66 (Official Build) (64-bit), my chromedriver version is 87.0.4280.88. And I'm using 64-bit Centos8 to run this on.
For example, take my two pieces of code I was testing, and see the folder structure as a result in the attached image (note the difference in headless=True):
self.driver_options = Options()
self.driver_options.headless = True
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.headless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
self.driver_options = Options()
self.driver_options.add_argument("--user-data-dir=/home/<user>/Projects/.nonheadless")
self.driver_options.add_argument("--window-size=1200,1200")
self.driver = webdriver.Chrome('../bin/chromedriver',options=self.driver_options)
Here's a picture comparing directory structure of headless (left) and non-headless (right)
I read that this may have something to do with headless chrome instances using a different data directory, so maybe some of the files are 'missing' due to that (in another location) - but the biggest issue is that the [user-dir]/Default/Cookies file is populated in the non-headless version, but unpopulated in the headless version. I've also tried to remedy it by adding a "remote-debugging-port" option, but still got the same outcome.
I want to run a few different headless threads, where cookies are important for long running jobs. Has anyone seen anything like this, or know why I am experiencing this behavior? Is it to be expected?
Thanks all