I'm trying to get Department names from this Walmart link. You can see that, firstly there are 7 departments on the left inside Departments (Chocolate cookies, cookies, butter cookies,...). When I click See All Departments , 9 more categories are added so now the number is 16. I'm trying to get all 16 departments with automatically. I wrote this code;
from selenium import webdriver
n_links = []
driver = webdriver.Chrome(executable_path='D:/Desktop/demo/chromedriver.exe')
url = "https://www.walmart.com/browse/snacks-cookies-chips/cookies/976759_976787_1001391"
driver.get(url)
search = driver.find_element_by_xpath("//*[@id='Departments']/div/div/ul").text
driver.find_element_by_xpath("//*[@id='Departments']/div/div/button/span").click()
search2 = driver.find_element_by_xpath("//*[@id='Departments']/div/div/div/div").text
sep = search.split('\n')
sep2 = search2.split('\n')
lngth = len(sep)
lngth2 = len(sep2)
for i in range (1,lngth):
path = "//*[@id='Departments']/div/div/ul/li"+"["+ str(i) + "]/a"
nav_links = driver.find_element_by_xpath(path).get_attribute('href')
n_links.append(nav_links)
for i in range (1,lngth2):
path = "//*[@id='Departments']/div/div/div/div/ul/li"+"["+ str(i) + "]/a"
nav_links2 = driver.find_element_by_xpath(path).get_attribute('href')
n_links.append(nav_links2)
print(n_links)
print(len(n_links))
When I run the code, in the end, I can see the links inside n_links array. But the problem is; sometimes it has 13 links, sometimes 14. It should be 16 and I haven't seen 16 yet, only 13 or 14. I tried to add time.sleep(3) before search2 line, but didn't work. Can you help me ?