How to use loop 'find next sibling' until reaching a certain tag when web scraping with beautifulsoup in python?

user_19007260 0
Viewed 71

The webpage I'm attempting to scrape has a section where the html tags are nested like so:

<div>
  <h3>
  <p>
  <p>
  <h3>
  <p>
  <p>
  <p>

My code is able to navigate to the correct tag but I am struggling to split the text by as

is a sibling, not a child. I am either able to print just the tags or print all the text within the tag without splitting into sections.

I've tried using for loops but I don't think is the right approach if searching within siblings. I think looping an if statement to determine if find_next_sibling().name = 'h3' might work but I've been unable to iterate this without nesting a large number of if statements.

Can anyone please advise on what approach I should take? Please see my full code below - the treaty files section works fine.

from bs4 import BeautifulSoup
import requests

url = 'https://www.gov.uk/government/publications/albania-tax-treaties'
get_url = requests.get(url)
url_html = get_url.content
soup = BeautifulSoup(url_html, 'lxml')

treaty_files = soup.find_all('div', class_='attachment-details')
for treaty_file in treaty_files:
    file_name = treaty_file.h3.a.text
    file_url = treaty_file.h3.a['href']
    #print(f"Treaty Name: {file_name}")
    #print(f"Treaty URL: {file_url}")
    #print()

#Attempt 1
treaty_details = soup.find('div', class_='govspeak').find_all('h3')
for treaty_content in treaty_details:
    content = treaty_content.find_next_siblings()
    for x in content:
        test = x
        a = test
        #print(a)

#Attempt 2
treaty_details = soup.find('div', class_='govspeak').find_all('h3')
for treaty_content in treaty_details:
    content = treaty_content.find_next_sibling()
    while content.name != 'h3':
        print(f"Text: {content.text}")
        content = content.find_next_sibling()
        if content.name == 'h3':
            break
web-scraping html python loops
1 Answers

One possible solution is to leverage pandas.Series.groupby function to group sections together:

import requests
import pandas as pd
from bs4 import BeautifulSoup


url = "https://www.gov.uk/government/publications/albania-tax-treaties"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

govspeak = soup.select_one(".govspeak")

s = pd.Series(govspeak.find_all(recursive=False))

for _, g in s.groupby(s.apply(lambda x: x.name).eq("h3").cumsum()):
    title = g.iloc[0].text
    text = "\n".join(row.text for row in g.iloc[1:])

    print(title)
    print("-" * 120)
    print(text)
    print()
    print()

Prints:

2021 UK-Albania Synthesised text of the Multilateral Instrument and the 2013 Double Taxation Agreement — in force
------------------------------------------------------------------------------------------------------------------------
The 2013 UK-Albania Double Taxation Agreement has been modified by the Multilateral Instrument (MLI).
The modifications made by the Multilateral Instrument entered into force in:

the UK on 1 October 2018
Albania on 1 January 2021

They are effective in the UK from:

1 January 2021 for taxes withheld at source
1 April 2022 for Corporation Tax
6 April 2022 for Income Tax and Capital Gains Tax

They are effective in Albania from 1 July 2021.


2013 UK-Albania Double Taxation Agreement — in force
------------------------------------------------------------------------------------------------------------------------
The agreement entered into force on 30 December 2013.
It is effective in the UK from:

1 April 2014 for Corporation Tax
6 April 2014 for Income Tax and Capital Gains Tax

It is effective in Albania from 1 January 2014 for Income Tax and Capital Gains Tax.
user_100359850
Related