I'm trying to web scrape a webpage and store the data in a CSV file. But I can't seem to get my code to work

user_11725402 0
Viewed 224

I'm trying to web scrape from this link, https://www.premierleague.com/stats/top/players/goals?se=-1 And I don't seem to be successful.

Below is the code I tried.

from urllib.request import urlopen
from bs4 import BeautifulSoup
import requests
import csv`

url = "https://www.premierleague.com/stats/top/players/goals?se=-1"
html = urlopen(url)
bs = BeautifulSoup(html, 'html.parser')

#print(bs)

listings = []

for rows in bs.find_all("tr"):
 if("oddrow" in rows["class"]) or ("evenrow" in rows["class"]):
    name = rows.find("div", class_="playerName").a.get_text()
    country = rows.find_all("td")[1].get_text()
    goals = rows.find_all("td")[4].get_text()
    listings.append([name, country, goals])

with open("EPL_TEST.csv", 'a', encoding = 'utf-8') as toWrite:
    writer = csv.writer(toWrite)
    writer.writerows(listings)

print("Data Fetched")

This is the error I'm getting: C:\Users\Siddhardh\Desktop\Python\Projects\FinalProject\venv\Scripts\python.exe C:/Users/Siddhardh/Desktop/Python/Projects/FinalProject/Scraping.py Traceback (most recent call last): File "C:/Users/Siddhardh/Desktop/Python/Projects/FinalProject/Scraping.py", line 16, in <module> if("oddrow" in rows["class"]) or ("evenrow" in rows["class"]): File "C:\Users\Siddhardh\Desktop\Python\Projects\FinalProject\venv\lib\site-packages\bs4\element.py", line 1016, in __getitem__ return self.attrs[key] KeyError: 'class'

Process finished with exit code 1

I need to get the names, country and goals of all the players into a CSV file.

P.S. Pardon my editing skills. This is my first post here. I'll learn.

web-scraping python
2 Answers

It looks like you have to change the middle section of your code to this:

listings = []

names = bs.find_all("td",scopr="row")
countries = bs.find_all("span",  {"class": "playerCountry"})
goals = bs.find_all("td",class_="mainStat")
for name, country, goal in zip(names,countries,goals):    
    listings.append([name.text.strip(), country.text.strip(), goal.text.strip()])

Printing out listings results in this output:

['Alan Shearer', 'England', '260'] ['Wayne Rooney', 'England', '208'] ['Andrew Cole', 'England', '187']

etc.

user_94480900

Try the script below to get all the names traversing multiple pages along with a data ridden csv file. You can use chrome dev tools to get the link I've used within the script. Using that link you will get json response. Modify to get all the other fields.

import csv
import requests
from bs4 import BeautifulSoup

url = "https://footballapi.pulselive.com/football/stats/ranked/players/goals?page={}&pageSize=20&comps=1&compCodeForActivePlayer=EN_PR&altIds=true"

headers = {
    'Origin': 'https://www.premierleague.com',
}

def get_items(link,page):
    while True:
        res = requests.get(link.format(page),headers=headers)
        soup = BeautifulSoup(res.text,"lxml")
        if not len(res.json()['stats']['content']):break
        for item in res.json()['stats']['content']:
            player_name = item['owner']['name']['display']
            yield player_name

        page+=1

if __name__ == '__main__':
    page = 112
    with open("player_info.csv","w", newline="") as outfile:
        writer = csv.writer(outfile)
        writer.writerow(['page','player'])
        for name in get_items(url,page):
            writer.writerow([name])

I used page = 112 to get all the names from that page onward. Feel free to make it 0 to get the names from start to the end.

user_91897990
Related