Web Scraping Wikipedia With BeautifulSoup

user_16848463 0
Viewed 49

I'm just trying to scrape the title of an input Wikipedia page but I get Bad Title after running

Here's the code

import requests
from bs4 import BeautifulSoup

wiki = input('Enter the wiki:')

url = "https://en.wikipedia.org/wiki/{wiki}"
response = requests.get(url)

soup = BeautifulSoup(response.content, 'html.parser')

title = soup.find(id="firstHeading")
print(title.string)

I wanted to eventually summaries the whole page but cant seem to get past printing the title

web-scraping beautifulsoup python
1 Answers

You will need to prefix your string with f for it to parse {wiki} correctly. You will probably also need to encode the URL to handle special characters correctly e.g. spaces. For example:

import urllib.parse        
import requests
from bs4 import BeautifulSoup

wiki = input('Enter the wiki:')

url = f"https://en.wikipedia.org/wiki/{urllib.parse.quote(wiki)}"
response = requests.get(url)

soup = BeautifulSoup(response.content, 'html.parser')

title = soup.find(id="firstHeading")
print(title.string)
user_49857330
Related