Webscraping into the desired python format

user_15861871 0
Viewed 35

I am trying to scrape a table from wikipedia using Beautifulsoup into my python notebook but I am unable to get my desired result.

I'm trying to convert the wikipedia table here enter image description here to this format in python enter image description here

I have tried the python code below code but it is not giving me the required output. How else can I achieve this please?

source = requests.get('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M').text

soup=BeautifulSoup(source,'lxml')

print(soup.title)

from IPython.display import display_html

tab = str(soup.table)

display_html(tab,raw=True)
beautifulsoup python
1 Answers

Try:

import re
import requests
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

for p in soup.select("td > p"):
    text = p.get_text(strip=True, separator=" ")
    post_code, borough, neighbourhood = re.search(
        r"^(M[^\s]+)\s*([^(]+)(?:\s*(.*))?", text
    ).groups()
    borough = borough.strip()
    neighbourhood = (neighbourhood or "Not Assigned").strip("() ")
    neighbourhood = neighbourhood.replace("(", "/").replace(")", "/")
    print("{:<5} {:<30} {}".format(post_code, borough, neighbourhood))

Prints:

M1A   Not assigned                   Not Assigned
M2A   Not assigned                   Not Assigned
M3A   North York                     Parkwoods
M4A   North York                     Victoria Village
M5A   Downtown Toronto               Regent Park / Harbourfront
M6A   North York                     Lawrence Manor / Lawrence Heights
M7A   Queen's Park                   Ontario Provincial Government
M8A   Not assigned                   Not Assigned

...and so on.

EDIT: To create panda's dataframe:

import re
import requests
import pandas as pd
from bs4 import BeautifulSoup

url = "https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

data = []
for p in soup.select("td > p"):
    text = p.get_text(strip=True, separator=" ")
    post_code, borough, neighbourhood = re.search(
        r"^(M[^\s]+)\s*([^(]+)(?:\s*(.*))?", text
    ).groups()
    borough = borough.strip()
    neighbourhood = (neighbourhood or "Not Assigned").strip("() ")
    neighbourhood = neighbourhood.replace("(", "/").replace(")", "/")

    data.append((post_code, borough, neighbourhood))

df = pd.DataFrame(data, columns=["Postcode", "Borough", "Neighborhood"])
print(df)
user_100359850
Related