Selecting everything between two heading anchor tags

user_5991208 0
Viewed 54

I am currently attempting to scrape a webpage for a small project. However, when attempting to select everything between two headings I've run into a problem. Here is an example of one of the headings I am trying to select:

<h2><a class="autolink" href="/compendium/dnd5e/Social%20Interaction#h-Long%20Rest">Long Rest</a></h2>

Found in: https://roll20.net/compendium/dnd5e/Rules:Resting/#h-Resting

And here is the code I have so far:

 els = soup.select("h2:has(a:contains('Short Rest')) ~ *:has(~ h2:has(a:contains('Long Rest')))")

However, using this returns a list of all the tags used and their, not the text as I thought. The desired output would be all the HTML between h2 'Short Rest' and h2 'Long Rest', so essentially the text describing a short rest.

The reason I am not using a simpler tag.next_sibling method is that there is a lot of untagged text on this webpage, which that method skips over.

Any help would be greatly appreciated, I've been stuck on this for a while now.

beautifulsoup css python
1 Answers

If I understand you correctly, this should do the trick:

import requests
from bs4 import BeautifulSoup as bs
from bs4 import NavigableString
soup = bs(req.text,'lxml')
final_text = []
for item in soup.find("h2",text="Short Rest").next_siblings:
    if item.name=="h2":
        break
    if isinstance(item, NavigableString):        
        final_text.extend(item)
    else:
        final_text.extend(item.stripped_strings)

print("".join(final_text))

Output:

A Short Rest is a period of downtime, at least 1 hour long, during which a character does nothing more strenuous than eating, drinking, reading, and tending to wounds.A character can spend one or more Hit Dice at the end of a Short Rest, up to the character’s maximum number of Hit Dice, which is equal to the character’s level. For each Hit Die spent in this way, the player rolls the die and adds the character’s Constitution modifier to it. The character regains Hit Points equal to the total. The player can decide to spend an additional Hit Die after each roll. A character regains some spent Hit Dice upon finishing a Long Rest, as explained below.

user_94480900
Related