scrape with BS4 Wikipedia text (pair each heading with paragraphs associated) - and output it to CSV-format

user_525865 0
Viewed 84

i want to pair web scraped paragraphs with the most recent scraped heading from wikipedia: as an example i have chosen the following wikipedia-article: https://en.wikipedia.org/wiki/England. I am currently scraping Wikipedia pages to find each paragraph, however, I am also scraping all the headings so I can put both together. I am trying to pair each heading with the paragraphs associated, however, i want to write it to a csv file. Note: we have the following h2 headings (of the paragraphs):

Toponymy
History
Governance
Geography
Economy
Healthcare

with that i can add the scraped text of another wikipedia text (eg. italy, spain, france etc.) to the csv - since i use the same headings. BTW: I am not sure if what i need is clear so feel free to ask questions.

the approach: the code I am using: With the following code.find_all('h2') tag and then use find_next_siblings('p') to get p tag after h2 until the next h2 find.

from bs4 import BeautifulSoup
import requests

page_link = 'https://en.wikipedia.org/wiki/England'
page_response = requests.get(page_link,verify=False, timeout=5)
page_content = BeautifulSoup(page_response.content, "html.parser")
textContent = []
for tag in page_content.find_all('h2')[1:]:
    texth2=tag.text.strip()
    textContent.append(texth2)
    for item in tag.find_next_siblings('p'):
        if texth2 in item.find_previous_siblings('h2')[0].text.strip():
            textContent.append(item.text.strip())
            
print(textContent)

see the output below:

btw: We have the following h2 headings (of the paragraphs)

Toponymy
History
Governance
Geography
Economy
Healthcare
Demography

and so on and so forth

all of the headings i want to write columns

def write_csv_file(content_list): with open ,,,, as csv_file: writer = csv.writer(csv_file, delimiter=',') writer.writerows(content_list)

Console output:

 ['Toponymy', 'The name "England" is derived from the Old English name Englaland, which means "land of the Angles".[15] The Angles were one of the Germanic tribes that settled in Great Britain during the Early Middle Ages. The Angles came from the Anglia peninsula in the Bay of Kiel area (present-day German state of Schleswig–Holstein) of the Baltic Sea.[16] The earliest recorded use of the term, as "Engla londe", is in the late-ninth-century translation into Old English of Bede\'s Ecclesiastical History of the English People. The term was then used in a different sense to the modern one, meaning "the land inhabited by the English", and it included English people in what is now south-east Scotland but was then part of the English kingdom of Northumbria. The Anglo-Saxon Chronicle recorded that the Domesday Book of 1086 covered the whole of England, meaning the English kingdom, but a few years later the Chronicle stated that King Malcolm III went "out of Scotlande into Lothian in Englaland", thus using it in the more ancient sense.[17]', 'The earliest attested reference to the Angles occurs in the 1st-century work by Tacitus, Germania, in which the Latin word Anglii is used.[18] The etymology of the tribal name itself is disputed by scholars; it has been suggested that it derives from the shape of the Angeln peninsula, an angular shape.[19] How and why a term derived from the name of a tribe that was less significant than others, such as the Saxons, came to be used for the entire country and its people is not known, but it seems this is related to the custom of calling the Germanic people in Britain Angli Saxones or English Saxons to distinguish them from continental Saxons (Eald-Seaxe) of Old Saxony between the Weser and Eider rivers in Northern Germany.[20] In Scottish Gaelic, another language which developed on the island of Great Britain, the Saxon tribe gave their name to the word for England (Sasunn);[21] similarly, the Welsh name for the English language is "Saesneg". A romantic name for England is Loegria, related to the Welsh word for England, Lloegr, and made popular by its use in Arthurian legend. Albion is also applied to England in a more poetic capacity,[22] though its original meaning is the island of Britain as a whole.', 'History', 'The earliest known evidence of human presence in the area now known as England was that of Homo antecessor, dating to approximately 780,000 years ago. The oldest proto-human bones discovered in England date from 500,000\xa0years ago.[23] Modern humans are known to have inhabited the area during the Upper Paleolithic period, though permanent settlements were only established within the last 6,000 years.[24][25]\nAfter the last ice age only large mammals such as mammoths, bison and woolly rhinoceros remained. Roughly 11,000\xa0years ago, when the ice sheets began to recede, humans repopulated the area; genetic research suggests they came from the northern part of the Iberian Peninsula.[26] The sea level was lower than now and Britain was connected by land bridge to Ireland and Eurasia.[27]\nAs the seas rose, it was separated from Ireland 10,000\xa0years ago and from Eurasia two millennia later.', 
    ....so on]

i want to write all the output into a csv-file - in other words - after scraping the Wikipedia text (pair each heading with paragraphs associated) - and output it to CSV-format

btw: with that i can add the scraped text of another wikipedia text (eg. italy, spain, france etc.) to the csv - since i use the same headings:

Toponymy
History
Governance
Geography
Economy
Healthcare
Demography

update: hi @Andrej Kesely: this is outstanding and so awesome - many many thanks; Btw;: can we apply this to the similar task too: see the collection of data for digital hubs shown here: https://s3platform.jrc.ec.europa.eu/digital-innovation-hubs-tool

with the hub-cards as data like the follwing:
https://s3platform-legacy.jrc.ec.europa.eu/digital-innovation-hubs-tool/-/dih/3480/view https://s3platform-legacy.jrc.ec.europa.eu/digital-innovation-hubs-tool/-/dih/13281/view https://s3platform-legacy.jrc.ec.europa.eu/digital-innovation-hubs-tool/-/dih/1417/view https://s3platform-legacy.jrc.ec.europa.eu/digital-innovation-hubs-tool/-/dih/1349/view

i have applied the scraper to this - and it works - but how to achieve a csv-output to the scraper that iterates on the urls: can we put the output into csv too - while applying the same technique!?

many thanks in advance.

web-scraping beautifulsoup csv html python
1 Answers

This example will get all <h2> as titles and following <p> as text from different URLs and saves it as CSV file:

import requests
import pandas as pd
from bs4 import BeautifulSoup
from itertools import accumulate, groupby


urls = [
    "https://en.wikipedia.org/wiki/England",
    "https://en.wikipedia.org/wiki/Portugal",
]

out = []
for url in urls:
    soup = BeautifulSoup(requests.get(url).content, "html.parser")

    tags = soup.select_one(".mw-parser-output").find_all(recursive=False)
    a = accumulate(t.name == "h2" for t in tags)

    d = {"Country": url.split("/")[-1]}
    for _, g in groupby(zip(a, tags), lambda k: k[0]):
        g = list(t for _, t in g if t.name in {"h2", "p"})
        if g[0].name != "h2" or len(g) == 1:
            continue

        title = g[0].get_text(strip=True).replace("[edit]", "")
        text = "\n".join(
            [t.get_text(strip=True, separator=" ") for t in g[1:]]
        ).strip()

        if not text:
            continue

        d[title] = text

    out.append(d)

df = pd.DataFrame(out)
df.to_csv("data.csv", index=False)

Saves data.csv (screenshot from Libre Office):

enter image description here

user_100359850
Related