Scrapy: CSS selector only extracting first two rows from table and I want every rows

user_10923739 0
Viewed 1133

I am trying to extract text from all td of a table from the below page. I am using CSS selector for this but somehow it not giving any output. I double-checked my CSS selector script in the browser inspect it works there but not in Scrapy.

HTML LInk: https://trusting-sinoussi-0dbf65.netlify.app/

enter image description here

In this HTML page I got two tables with same name, first one has nothing in it and second one has all data. And this is my scrapy code for selecting the tr and then extracting td's text. There is more than one table so I used the summary attribute to select the table. I am using for loop to extract td's from all the tr. So, in variable courses, I scraped till all tr and then passed that to for loop to extract td from each tr. but somehow output only shows for first tr only it is not selecting all the tr from this table

When I extract using

response.css('tr').extract()

OUTPUT includes only first two tr

But I want all td's text inside each tr so I tried to use td instead of tr

response.css('td.dddefault ::text').extract()

then it returns all the td from the table. **But I am not able to understand that when using tr as ectracting tag it only outputs first 2 rows values??? why? **

def course_scrap(self, response):
        print('course slected')
        courses = response.css('table.datadisplaytable[summary="This layout table is used to present the course found"] tr')

        for course in courses:
            trs = course.css('td.dddefault ::text').extract()

        print(trs)
scrape scrapy web-scraping python
2 Answers

To get the first actual row use:

table.datadisplaytable[summary="This layout table is used to present the course found"] tr:nth-child(3)

Second:

table.datadisplaytable[summary="This layout table is used to present the course found"] tr:nth-child(4)

First two tr's are headers.

The same approach can be used to get a shorter table locator:

.datadisplaytable:nth-of-type(2)
user_127301120

This should fetch you the numbers and their concerning texts out of that table from that site:

class TrustingSpider(scrapy.Spider):
    name = "trusting"
    start_urls = ['https://trusting-sinoussi-0dbf65.netlify.app/']

    def parse(self,response):
        for item in response.xpath("//table[@class='datadisplaytable'][./tbody]//tr[./td]"):
            td_first = item.xpath(".//td[@class='dddefault']/text()").get()
            td_second = item.xpath(".//td[@class='dddefault'][@text-align]/text()").get()
            yield {"Number":td_first,"Text":td_second}

First few results:

051 CS Co-op Work Term #1
052 CS Co-op Work Term #2
053 CS Co-op Work Term #3
054 CS Co-op Work Term #4
055 CS Co-op Work Term #5
100 Introduction to Computers
110 Prog & Problem Solving
115 Object-Oriented Design
201 Intro to Digital System
203 Java Programming and Apps
207 Building Interactive Gadgets
210 Data Structures & Abstractions
215 Web and Database Programming
261 Methods in Numerical Analysis
280 Risk and Reward in Information
310 Discrete Computal Structure
320 Intro Artificial Intelligence
user_91897990
Related