How to scrape wikipedia if you already have all the urls?

user_15179161 0
Viewed 72

I have the a table of information below that contains 1000 of these entries and the sum of the first column is approximately 90 000.

counts genus species
14149 Marchantia polymorpha
9345 Physcomitrium patens
7744 Selaginella moellendorffii
5389 Picea sitchensis

For each of the 1000 entries I would like to search the Wikipedia page and extract the grouping of the organism.

For example:

I look up Marchantia polymorpha in Wikipedia and find it's page. On the right most side of the page is the scientific classification of the species. I would like to extract the value for Kingdom i.e. Plantae (amongst others) in this case.

Wikipedia results page

At the end of searching and extracting I would like a table that looks like this:

counts genus species kingdom
14149 Marchantia polymorpha plantae
9345 Physcomitrium patens plantae
7744 Selaginella moellendorffii plantae
5389 Picea sitchensis fungi

So that I can count the total number of entries belonging to each kingdom.

Collecting the URLs won't be difficult. Since the base URL is the same and the page I want to end up on simply adds the genus_species name. For example all the URLs for the list above would be:

https://en.wikipedia.org/wiki/Marchantia_polymorpha  
https://en.wikipedia.org/wiki/Physcomitrium_patens  
https://en.wikipedia.org/wiki/Selaginella_moellendorffii  
https://en.wikipedia.org/wiki/Picea_sitchensis

I am not 100% certain that a Wikipedia page exists for each genus but the majority do based on manual searches I've done previously.

web-scraping r python
1 Answers

You can use this, but I am not sure if it will work on all Wikipedia sites:

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/wiki/Marchantia_polymorpha'
res = requests.get(url)
soup = BeautifulSoup(res.content, 'html.parser')
# It is in an table row:
tr = soup.find_all('tr')
for t in tr:
    if 'kingdom' in t.text.lower():
        text = t.text
        break
text.replace('\n', '').split(':')[1]
>>> 'Plantae'

# Or since both kingdom and plantea are in a <td>
td = soup.find_all('td')
for i, t in enumerate(td):
    if 'kingdom' in t.text.lower():
        kingdom = td[i+1].text.rstrip() # in the row after kingdom is plantae (i+1)

print(kingdom)
>>> 'Plantae'
user_57753580
Related