xpath join text from multiple elements python

Question

Hello I have some html file from this website: https://www.oddsportal.com/soccer/argentina/superliga/results/


  
    Independiente
    "- San Martin T."
  



  
    "Lanus - " 
    Huracan
  



  Rosario Central - Colon Santa FE

I want to select and join a/text() and span/text() in order to look like this: "Independiente - San Martin T." As you see the 'span' is not allways in the same place and some times is missing (see last 'td class')

I used this code:

('//td[@class="name table-participant"]/a/text() | span/text()').extract()

but it returns only the a/text(). Can you help me to make this work? Thank you

user_10250107 · Answer

You trying to search span/text() without a scope. Add // at the beginning of this part of query, in the totally:

('//td[@class="name table-participant"]/a/text() | //span/text()').extract()

But I'm strongly recommend use this decision:

 ('//td[@class="name table-participant"]//*[self::a/ancestor::td or self::span]/text()').extract

for get span only from your choiced td-scope.

user_2790838 · Answer

I'm assuming that you're using Scrapy to scrape the HTML.

From the structure of your sample HTML, it looks like you want to obtain the text of the anchor element, so you need to iterate over those.

Only then you can strip and join the text child nodes of the anchor element to obtain properly formatted strings. There is additional complication by the inconsistent use of quotes, but the following should get you going.

from scrapy.selector import Selector

HTML="""

  
    Independiente
    "- San Martin T."
  



  
    "Lanus - "
    Huracan
  



  Rosario Central - Colon Santa FE

"""

def strip_and_join(x):
    l=[]
    for s in x:
        # strip whitespace and quotes
        s = s.strip().strip('"').strip()
        # drop now empty strings
        if s:
            l.append(s)
    return " ".join(l)

for x in Selector(text=HTML).xpath('//td[@class="name table-participant"]/a'):
    print strip_and_join(x.xpath('.//text()').extract())

Note that for the sake of clarity I didn't squeeze the code into a single list comprehension, although this would be possible of course.

xpath join text from multiple elements python

2 Answers