Is it possible to scrape these specific href links?

Jherky 0
Viewed 32

I'm trying to scrape specific href links but I can only grab no links or every link possible on the website is it possible to grab every link that's within a div tag? the div tags have the same exact name but I can't find a way to look within the div tag then grab the href tags

here is the code that I am currently using is any work around on this?

    rom selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait, Select
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

# Establish chrome driver and go to report site URL
url = "https://navalcommand.enjin.com/forum/viewforum/2989694/m/11178354/page/1"
driver = webdriver.Chrome()
driver.get(url)

elems = driver.find_elements(By.XPATH, '//a')

for elem in elems:
    print(elem.get_attribute("href"))

the div tags that the web driver should be looking for is the structure small-cells tag and then the href tag if it is possible

web-scraping selenium python
1 Answers

All you need here is a correct locator.
You can use this CSS-SELECTOR .structure.small-cells a[href]
Or this XPATH "//table[@class='structure small-cells']//a[@href]"
They are similar, just different syntax.
So, your Selenium code can be as following:

elems = driver.find_elements(By.XPATH, "//table[@class='structure small-cells']//a[@href]")

for elem in elems:
    print(elem.get_attribute("href"))

To make this code work properly you will need to add a delay, preferably webdriverwaits expected conditions.
Something like this:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC


options = Options()
options.add_argument("start-maximized")


webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(service=webdriver_service, options=options)
url = "https://navalcommand.enjin.com/forum/viewforum/2989694/m/11178354/page/1"
driver.get(url)
wait = WebDriverWait(driver, 20)

elems = wait.until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@class='structure small-cells']//a[@href]")))

for elem in elems:
    print(elem.get_attribute("href"))

However this will probably not work since that site is blocking Selenium driven sessions...

Prophet0
Related