Having trouble extracting html data with xpath

user_19364322 0
Viewed 49

Currently I'm trying to extract the location data from specific IPs in talosintelligence. When I run this code, it returns nothing.

I'm not sure what I'm doing wrong here. In this case, I would prefer for it to return "Charlotte, United States". Instead I just get an empty list.

What am I doing wrong here? Any help is greatly appreciated!

#pip install lxml
#pip install requests

from lxml import html 
import requests

page = requests.get('https://talosintelligence.com/reputation_center/lookup?search=198.86.225.18')
tree = html.fromstring(page.content)

#gets talos ip location 
talosiploc = tree.xpath('//*[@id="location-data-wrapper"]/table/tr/td/text()')
print(talosiploc)
web-scraping python
1 Answers

I tried your request and got 403 response code forbidden.
This web use anti-bot security CloudFlare. How to bypass CloudFlare bot protection?
It depends.... Sometimes it is very difficult task or impossible.
That you need to do, is simulate a real user with the browser.
Try to use playwright, with pw you have 3 browsers: chromium, firefox and webkit.
Here you have some example of working code:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.webkit.launch(headless=True)
    baseurl = "https://talosintelligence.com/reputation_center/lookup?search=198.86.225.18"
    page = browser.new_page()
    page.goto(baseurl)
    title = page.wait_for_selector('//*[@id="location-data-wrapper"]/table/tr/td')
    print("Country: " + title.text_content())
    browser.close()

OUTPUT:

Country: Charlotte, United States

I hope I have been able to help you.

user_152135030
Related