Reaching tabs of content in a dynamic website

userOLlg1 0
Viewed 29

I am trying to scrape cars in Gumtree (https://www.gumtree.com/p/nissan/2017-nissan-juke-1.5-dci-tekna-pulse-euro-6-s-s-5dr-hatchback-diesel-manual/1437058688)

There is a part below with all the car specifications and I can't reach the other tabs:

Table

I have tried using Selenium in headless mode, beautifulSoup and a combination of both.

def getData(url):
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'lxml')
parentList = soup.find('div', {'class': 'css-1j2uvky e14ksbtl9'})

for e in parentList:
    print(e)
    driver.implicitly_wait(5)
    data = soup.find('ul', {'class': 'css-kcx0xb e14ksbtl6'})
    for a in data:
        print(a.text)
    e.click()

Thank you!

beautifulsoup web-scraping selenium webdriver python
1 Answers

In BS4 find() returns only element not the list of elements, so you can't iterate it you need to use find_all().

You need to get the webelement to click and use driver.find_element() and get lists of elements use driver.find_elements()

code:

def getData(url):
  driver.get(url)
  driver.implicitly_wait(5)
  #Click on Cookie buuton
  driver.find_element(By.ID, "onetrust-accept-btn-handler").click()
  #Get all tabs element
  parentList = driver.find_elements(By.XPATH, "//h4[@data-q='motors-attributes-titles']")

  for e in parentList:
    print("Tab : " +e.text)
    driver.execute_script("arguments[0].click();", e)
    driver.implicitly_wait(5)
    soup = BeautifulSoup(driver.page_source, 'lxml')    
    
    data = soup.find_all('ul', {'class': 'css-kcx0xb e14ksbtl6'})
    for a in data:
        print(a.text)
    print("=======================================")

getData("https://www.gumtree.com/p/nissan/2017-nissan-juke-1.5-dci-tekna-pulse-euro-6-s-s-5dr-hatchback-diesel-manual/1437058688")

Output:

Tab : Overview
Year2017Mileage47,000 milesBody TypeHatchbackTransmissionManualColourBlackSeats5Doors5Luggage Capacity (seats Up)354 litres
=======================================
Tab : Performance
Fuel TypeDieselEngine Power108.6 bhpEngine Size1461 ccBrochure Engine Size1.5 LTop Speed109 mphAcceleration (0-62mph)11.2 secondsYear2017Mileage47,000 milesBody TypeHatchbackTransmissionManualColourBlackSeats5Doors5Luggage Capacity (seats Up)354 litres
=======================================
Tab : Running Costs
Fuel Consumption70.6 mpgFuel Capacity46 litresUrban Mpg64.2 mpgExtra Urban Mpg74.3 mpgInsurance Group12ECO2 Emissions104 g/kmEuro EmissionsEuro 6Fuel TypeDieselEngine Power108.6 bhpEngine Size1461 ccBrochure Engine Size1.5 LTop Speed109 mphAcceleration (0-62mph)11.2 secondsYear2017Mileage47,000 milesBody TypeHatchbackTransmissionManualColourBlackSeats5Doors5Luggage Capacity (seats Up)354 litres
=======================================
Tab : Safety & Security
=======================================
Tab : Driving Convenience
=======================================
Tab : Exterior
=======================================
KunduK0
Related