Can't create an xpath capable of meeting certain condition

user_10568531 0
Viewed 63

I've created a script which is able to extract the links ending with .html extention available under class tableFile from a webpage. The script can do it's job. However, my intention at this point is to get only those .html links which have EX- in its type field. I'm looking for any pure xpath solution (by not using .getparent() or something).

Link to that site

Script I've tried with so far:

import requests
from lxml.html import fromstring

res = requests.get("https://www.sec.gov/Archives/edgar/data/1085596/000146970918000185/0001469709-18-000185-index.htm")
root = fromstring(res.text)

for item in root.xpath('//table[contains(@summary,"Document")]//td[@scope="row"]/a/@href'):
    if ".htm" in item:
        print(item)

When I try to get the links meeting above condition with the below approach, I get an error:

for item in root.xpath('//table[contains(@summary,"Document")]//td[@scope="row"]/a/@href'):
    if ".htm" in item and "EX" in item.xpath("..//following-sibling::td/text"):
        print(item)

Error I get:

if ".htm" in item and "EX" in item.xpath("..//following-sibling::td/text"):
AttributeError: 'lxml.etree._ElementUnicodeResult' object has no attribute 'xpath'

This is how the files look like:

enter image description here

web-scraping python-3.x xpath python
3 Answers

If you need pure XPath solution, you can use below:

import requests
from lxml.html import fromstring

res = requests.get("https://www.sec.gov/Archives/edgar/data/1085596/000146970918000185/0001469709-18-000185-index.htm")
root = fromstring(res.text)
for item in root.xpath('//table[contains(@summary,"Document")]//tr[td[starts-with(., "EX-")]]/td/a[contains(@href, ".htm")]/@href'):
    print(item)


/Archives/edgar/data/1085596/000146970918000185/ex31_1apg.htm
/Archives/edgar/data/1085596/000146970918000185/ex31_2apg.htm
/Archives/edgar/data/1085596/000146970918000185/ex32_1apg.htm
/Archives/edgar/data/1085596/000146970918000185/ex32_2apg.htm
user_45495540

Here is a way using dataframes and pandas

import pandas as pd
tables = pd.read_html("https://www.sec.gov/Archives/edgar/data/1085596/000146970918000185/0001469709-18-000185-index.htm")
base = "https://www.sec.gov/Archives/edgar/data/1085596/000146970918000185/"
results = [base + row[1][2] for row in tables[0].iterrows() if row[1][2].endswith(('.htm', '.txt')) and str(row[1][3]).startswith('EX')]
print(results)
user_62412350

It looks like you want:

//td[following-sibling::td[starts-with(text(), "EX")]]/a[contains(@href, ".htm")]

There's a lot of different ways to do this with xpath. Css is probalby much simpler.

user_9660230
Related