Extract text inside anchor tag using xpath

Viewed 1633

I am trying to ascertain how many pages are there for any search result on a site so that i can scrape data for all the pages using lxml and xpath.

There is a pagination tab with the following structure: Page: 1 2 3 ... 7 next

the html content for the same being something like

<ul class="ulclass">
 <li></li>
 <li>
      <span> You are on the first page</span>
      "1"
 </li>
 <li>
      <a href="link to second page">
        <span></span>
      "2"
      </a>
 </li>
  <li>
 </li>
      ...
  <li>
      <a href="link to last page">
        <span></span>
      "7"
      </a>
 </li>

My approach is to extract the page numbers 1,2,3,7 so that i can repeat the web scraping 7 times for every page 'cause otherwise it just scrapes the first result of the page. I have written the following xpath, but it doesnot return correct page numbers.

xpath('//ul[@class="ulclass"]/li/a/text())

1 Answers
Related