Web scraping multiple similar pages

user_13290733 0
Viewed 77

I'm new to python web scraping, and I was attempting to take the addresses of different winmar locations in Canada, as well as put the results into a csv file. So far, the only way I have found to differentiate between the different locations' sites is by a code at the end of the address (numbers). The problem is that the results do not change as the program runs, and instead produces the results of the first location (305) when printing and into the csv file. Thanks for your time and consideration!

Here's my code:

import csv
import requests
from bs4 import BeautifulSoup

x = 0
numbers = ['305', '405', '306', '307', '308', '309', '4273']

f = csv.writer(open('Winmar_locations.csv', 'w'))
f.writerow(['City:', 'Address:'])

for links in numbers:

    for x in range(0, 6):
        url = 'https://www.winmar.ca/find-a-location/' + str(numbers[x])
        r = requests.get(url)
        soup = BeautifulSoup(r.content, "html.parser")

    location_name = soup.find("div", attrs={"class": "title_block"})
    location_name_items = location_name.find_all('h2')

    location_list = soup.find(class_='quick_info')
    location_list_items = location_list.find_all('p')

    for name in location_name_items:
        names = name.text
        names = names.replace('Location | ', '')

    for location in location_list_items:
        locations = location.text.strip()
        locations = locations.replace('24 Hour Emergency | (902) 679-1116','')

    print(names, locations)
    x = x+1

    f.writerow([names, locations])
web-scraping beautifulsoup python
2 Answers

You had a few things wrong in your code and one thing about the website you are scraping

  • First accessing the url like this https://www.winmar.ca/find-a-location/308 will not change the location properly, it needs to be like this https://www.winmar.ca/find-a-location/#308 with hashbang before the number.

  • The website has duplicate html with the same classes, that means you nearly have all locations loaded all the time and they just choose which to show from their js code -bad practice ofcourse-, that makes your matcher always gets the same location, which explains why you always had the same location repeated.

  • Lastly, you had a lot of unnecessary loops, you only need to loop over the numbers array and that's it.

here is a modified version of your code

import csv
import requests
from bs4 import BeautifulSoup

x = 0
numbers = ['305', '405', '306', '307', '308', '309', '4273']


names = []
locations = []
for x in range(0, 6):
    url = 'https://www.winmar.ca/find-a-location/#' + str(numbers[x])
    print(f"pinging url {url}")

    r = requests.get(url)
    soup = BeautifulSoup(r.content, "html.parser")
    scope = soup.find(attrs={"data-id": str(numbers[x])})

    location_name = scope.find("div", attrs={"class": "title_block"})
    location_name_items = location_name.find_all('h2')


    location_list = scope.find(class_='quick_info')
    location_list_items = location_list.find_all('p')

    name = location_name.find_all("h2")[0].text
    print(name)

    names.append(name)

    for location in location_list_items:
        loc = location.text.strip()
        if '24 Hour Emergency' in loc: 
            continue
        print(loc)
        locations.append(loc)

    x = x+1

Notice the scoping I did

    scope = soup.find(attrs={"data-id": str(numbers[x])})

that makes your code immune to how much locations they have loaded in the html, you only targeting the scope with the location you want.

this results in :

pinging url https://www.winmar.ca/find-a-location/#305
Location | Annapolis
70 Donald E Hiltz Connector Road
Kentville, NS
B4N 3V7
pinging url https://www.winmar.ca/find-a-location/#405
Location | Bridgewater
15585 Highway # 3
Hebbville, NS
B4V 6X7
pinging url https://www.winmar.ca/find-a-location/#306
Location | Halifax
9 Isnor Dr
Dartmouth, NS
B3B 1M1
pinging url https://www.winmar.ca/find-a-location/#307
Location | New Glasgow
5074 Hwy. #4, RR #1
Westville, NS
B0K 2A0
pinging url https://www.winmar.ca/find-a-location/#308
Location | Port Hawkesbury
8 Industrial Park Rd
Lennox Passage, NS
B0E 1V0
pinging url https://www.winmar.ca/find-a-location/#309
Location | Sydney
358 Keltic Drive
Sydney River, NS
B1R 1V7
user_64842760

Although you have got a qualified answer, I thought to come up with mine. I've tried to make the script concise shaking off verbosity. Make sure your bs4 version is 4.7.0 or later in order for it to support pseudo selector which I've defined within the script to locate the address.

import csv
import requests
from bs4 import BeautifulSoup

base = 'https://www.winmar.ca/find-a-location/#{}'

numbers = ['305', '405', '306', '307', '308', '309', '4273']

with open("Winmar_locations.csv","w",newline="") as f:
    writer = csv.writer(f)
    writer.writerow(['City','Address'])

    while numbers:
        num = numbers.pop(0)
        r = requests.get(base.format(num))
        soup = BeautifulSoup(r.content,"html.parser")

        location_name = soup.select_one(f"[data-id='{num}'] .title_block > h2.title").contents[-1]
        location_address = soup.select_one(f"[data-id='{num}'] .heading:contains('Address') + p").get_text(strip=True)
        writer.writerow([location_name,location_address])
        print(location_name,location_address)
user_129651960
Related