New to Python and Web-scraping. Scraping an HTML table- however it's not displaying all columns

user_19004470 0
Viewed 35

I'm using BeautifulSoup and trying to scrap a HTML table. I'm only interested in the first table. However, the output is missing one column values - the "Entries" column. Not sure what I'm doing wrong here.

Here's my code:

import requests
from bs4 import BeautifulSoup

URL = "http://www.godaycare.com/child-care-cost/saskatchewan"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

table = soup.find_all('table')[0]

for child in soup.find_all('table')[0].children:
    for td in child:
        print(td.text)

Here's the output

TypeAge Cat.SpotAVG. Cost ($)Entries
LicensedInfantFull-Time751.02717
LicensedInfantPart-Time41.31187
UnlicensedInfantFull-Time699.56287
UnlicensedInfantPart-Time31.0550
LicensedToddlerFull-Time661.04604
LicensedToddlerPart-Time32.69148
UnlicensedToddlerFull-Time633.01342
UnlicensedToddlerPart-Time35.9969
LicensedPreschoolFull-Time595.45327
LicensedPreschoolPart-Time30.8566
UnlicensedPreschoolFull-Time602.82195
UnlicensedPreschoolPart-Time30.3330
LicensedKindergartenFull-Time562.8787
LicensedKindergartenPart-Time28.2938
UnlicensedKindergartenFull-Time549.1257
UnlicensedKindergartenPart-Time23.0113
LicensedSchoolageFull-Time605.3494
LicensedSchoolagePart-Time25.4533
UnlicensedSchoolageFull-Time434.9098
UnlicensedSchoolagePart-Time19.0025
web-scraping beautifulsoup python
1 Answers

The easiest way how to read the first table is using pandas.read_html:

import pandas as pd

url = "http://www.godaycare.com/child-care-cost/saskatchewan"

df = pd.read_html(url)[0]
print(df.to_markdown())

Prints:

Type Age Cat. Spot AVG. Cost ($) Entries
0 Licensed Infant Full-Time 751.02 717
1 Licensed Infant Part-Time 41.31 187
2 Unlicensed Infant Full-Time 699.56 287
3 Unlicensed Infant Part-Time 31.05 50
4 Licensed Toddler Full-Time 661.04 604
5 Licensed Toddler Part-Time 32.69 148
6 Unlicensed Toddler Full-Time 633.01 342
7 Unlicensed Toddler Part-Time 35.99 69
8 Licensed Preschool Full-Time 595.45 327
9 Licensed Preschool Part-Time 30.85 66
10 Unlicensed Preschool Full-Time 602.82 195
11 Unlicensed Preschool Part-Time 30.33 30
12 Licensed Kindergarten Full-Time 562.87 87
13 Licensed Kindergarten Part-Time 28.29 38
14 Unlicensed Kindergarten Full-Time 549.12 57
15 Unlicensed Kindergarten Part-Time 23.01 13
16 Licensed Schoolage Full-Time 605.34 94
17 Licensed Schoolage Part-Time 25.45 33
18 Unlicensed Schoolage Full-Time 434.9 98
19 Unlicensed Schoolage Part-Time 19 25

EDIT: Version with beautifulsoup:

import requests
from bs4 import BeautifulSoup

URL = "http://www.godaycare.com/child-care-cost/saskatchewan"
page = requests.get(URL)

soup = BeautifulSoup(page.content, "html.parser")

for row in soup.find("table").find_all("tr"):
    tds = [td.text for td in row.find_all(["td", "th"])]
    print(("{:<20}" * len(tds)).format(*tds))

Prints:

Type                Age Cat.            Spot                AVG. Cost ($)       Entries             
Licensed            Infant              Full-Time           751.02              717                 
Licensed            Infant              Part-Time           41.31               187                 
Unlicensed          Infant              Full-Time           699.56              287                 
Unlicensed          Infant              Part-Time           31.05               50                  
Licensed            Toddler             Full-Time           661.04              604                 
Licensed            Toddler             Part-Time           32.69               148                 
Unlicensed          Toddler             Full-Time           633.01              342                 
Unlicensed          Toddler             Part-Time           35.99               69                  
Licensed            Preschool           Full-Time           595.45              327                 
Licensed            Preschool           Part-Time           30.85               66                  
Unlicensed          Preschool           Full-Time           602.82              195                 
Unlicensed          Preschool           Part-Time           30.33               30                  
Licensed            Kindergarten        Full-Time           562.87              87                  
Licensed            Kindergarten        Part-Time           28.29               38                  
Unlicensed          Kindergarten        Full-Time           549.12              57                  
Unlicensed          Kindergarten        Part-Time           23.01               13                  
Licensed            Schoolage           Full-Time           605.34              94                  
Licensed            Schoolage           Part-Time           25.45               33                  
Unlicensed          Schoolage           Full-Time           434.90              98                  
Unlicensed          Schoolage           Part-Time           19.00               25
user_100359850
Related