How do I use xpath to obtain the text between the two h2 tags?

Question

I'm building a web scraper that gathers the names from a website and then compiles them into a csv. I'm new to using scrapy and xpath but by all accounts, the code should work (isn't that the truth). Also, the code doesn't output a csv file with the data either.

Here's the html data:

Text I want to copy

Here's my code:

from distutils.filelist import findall
import email
from platform import python_branch
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
import re
import csv

engine = 'python'

filename = 'test.csv'

url_list = []
email_list = []

class EmailsSpider(CrawlSpider):
    
    name = 'emails'

    # change the below URLS

    allowed_domains = ['my allowed domain(holder)']
    start_urls = ['my start url (holder)']

    rules = (
        Rule(LinkExtractor(allow=''), callback='parse_item', follow=True),
    )

    def parse_item(self, response):
        names = response.xpath("//h2[@class='et_pb_module_header'] //text()[normalize-space()] ").getall()
        

        with open(filename, 'w') as csvfile: 
            # creating a csv writer object 
            csvwriter = csv.writer(csvfile) 
                
            # writing the fields 
            csvwriter.writerow(names)

I'm not really sure what's going on and would greatly appreciate any advice or direction. Thanks!

I've tried the link below along with many other sites too to no avail: Extract all Text from child elements/node using xpath text() function

Here is the website with the names that I'm trying to scrape, I am trying to compile the list of names: https://counseling.uci.edu/about-us/meet-our-staff/directors/

How do I use xpath to obtain the text between the two h2 tags?

0 Answers