Getting death date of a person from a wikipedia link using BeautifulSoup

user_10066068 0
Viewed 91

I'm trying to scrape the dates a person died from a bunch of wikipedia links I have in a csv filen and save it as another column. It has both people who are alive currently and those who have died. How do I scrape this data for the people who have a died date in the wikipedia article?

Below is the code that I'm using to get the dates

from bs4 import BeautifulSoup
import requests
import pandas as pd

df = pd.read_csv('wiki_links.csv')

#print(df.head())

for index, row in df.iterrows():
    req= requests.get(row['url'])
    soup = BeautifulSoup(req.content, features='lxml')
    dates = soup.select('table.infobox td.infobox-data span')
    for ddate in dates:
        print(ddate)
        #df.at[index, 'date'] = ddate.string
    print()   
    
#df.to_csv('Scraped/wiki_links.csv', index=0)

The sample output of the following code for both cateories are as follows

https://en.wikipedia.org/wiki/Lewis_Hamilton

<span style="display:none"> (<span class="bday">1985-01-07</span>) </span>
<span class="bday">1985-01-07</span>
<span class="noprint ForceAgeToShow"> (age 36)</span>
<span class="flagicon"><a href="/wiki/United_Kingdom" title="United Kingdom"><img alt="United Kingdom" class="thumbborder" data-file-height="600" data-file-width="1200" decoding="async" height="12" src="//upload.wikimedia.org/wikipedia/en/thumb/a/ae/Flag_of_the_United_Kingdom.svg/23px-Flag_of_the_United_Kingdom.svg.png" srcset="//upload.wikimedia.org/wikipedia/en/thumb/a/ae/Flag_of_the_United_Kingdom.svg/35px-Flag_of_the_United_Kingdom.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/a/ae/Flag_of_the_United_Kingdom.svg/46px-Flag_of_the_United_Kingdom.svg.png 2x" width="23"/></a></span>
<span class="url"><a class="external text" href="http://lewishamilton.com/" rel="nofollow">lewishamilton<wbr/>.com</a></span>

https://en.wikipedia.org/wiki/James_Hunt

<span style="display:none">(<span class="bday">1947-08-29</span>)</span>
<span class="bday">1947-08-29</span>
<span style="display:none">(1993-06-15)</span>

How do I get the date the person died and save it in the csv while keeping it blank for for those who are still alive?

pandas dataframe beautifulsoup python
1 Answers
from bs4 import BeautifulSoup
import requests
import pandas as pd

df = pd.read_csv('wiki_links.csv')

#print(df.head())

for index, row in df.iterrows():
    req= requests.get(row['url'])
    soup = BeautifulSoup(req.content, features='lxml')
    dates = soup.select('table.infobox td.infobox-data span')
    for date in dates:
      if date.text == 'Died':
        ddate = date.find_next('td',{'class':'infobox-data'},recursive=False).find_all(text=True,recursive=False)[0]
        print(ddate)
        #df.at[index, 'date'] = ddate.string
    print()   
    
#df.to_csv('Scraped/wiki_links.csv', index=0)
user_154520440
Related