How can I get HTML elements between two independent tags

user_11317290 0
Viewed 201

I am using puppeteer. I have a situation where I need to get content between two tags which are not in a parent-child relationship.

<h1>neverchangeA<h1>
<span>abc<span>
<span>abc2<span>
<h1>neverchangeB<h1>

Expected elements

<span>abc<span>
<span>abc2<span>

In simple, I need something like a regex similar to this:

regex.matchBetween(<h1>neverchangeA<h1>,<h1>neverchangeB<h1>)
puppeteer node.js web-scraping javascript
4 Answers

Getting the sibling of an elementHandle in Puppeteer explains how to get the previous sibling of an element with puppeteer. There is a similar function to get the next sibling of an element. You can apply this to your situation by writing a loop that starts with the first <h1> element then repeatedly gets the next sibling until you reach the second <h1> element.

user_14405650

You could do this with JS and the evaluate method.

https://github.com/puppeteer/puppeteer/blob/master/docs/api.md#pageevaluatepagefunction-args

This example returns the HTML of the desired elements as a string. 

const result = await page.evaluate(() => {
  const h1s = [...document.querySelectorAll('h1')]
  const neverChangeA = h1s.find(elem => elem.innerText === "neverchangeA")
  if(neverChangeA){
    const siblings = [...neverChangeA.parentNode.children]
    const indexOfFirstH1 = siblings.findIndex(elem => elem.innerText === "neverchangeA")
    const indexOfSecondH1 =  siblings.findIndex(elem => elem.innerText === "neverchangeB")
    const betweenELems = siblings.slice(indexOfFirstH1 + 1, indexOfSecondH1)
    const htmlOfElems = betweenELems.map(elem => elem.outerHTML)
    const result = htmlOfElems.join('')
    return Promise.resolve(result)
  }
  else {
    return Promise.resolve(null)
 }
})
console.log(result)
user_92422250

Solution using XPath

This is a good use case for XPath. The following query looks for span elements that have an h1 tag with content neverchangeA before them and an h1 tag with the content neverchangeB after them:

//span[preceding::h1="neverchangeA" and following::h1="neverchangeB"]

To use an XPath expression within puppeteer, use page.$x.

Code Sample

const spans = await page.$x('//span[preceding::h1="neverchangeA" and following::h1="neverchangeB"]');
user_56275990

You should use regex. This: <h1>.*<h1> will select the h1 tag and whatever there is between the tag. One way is to remove the result of this from the text, and you will have the result you need.

user_91373790
Related