How to improve Beautiful soup webscrape loop?

Question

I'm dipping my toes into webscraping with beautiful soup, and to do so I'm doing a small project where I'm looking at a pokemon fansite and getting the pokemon moves from a table. I'm going for the move name and nothing else. Currently my code does that poorly and incorrectly until the very bottom of its output. It looks something like this.

It eventually does what I anticipate at the end there (starting with pound).

Here is what the table looks like on the webpage.

What I've got:

import requests
from bs4 import BeautifulSoup as bs

# Load page
r = requests.get("https://bulbapedia.bulbagarden.net/wiki/List_of_moves")

# Convert to soup object
soup = bs(r.content)

# Get first table (aka the one we need)
first_table = soup.find('table')

# Loop and grab what we want
for td in first_table.find_all('td', style=False, align=False):
    download = td.find_all('a', href=True, title=True, style=False, align=False)
    for a in download:
        text = a.string
        print(text)
input()

user_12309808 · Answer

All of this is not even necessary. You can simply use pandas to scrape the entire table:

import requests
import pandas as pd

r = requests.get("https://bulbapedia.bulbagarden.net/wiki/List_of_moves")

df = pd.read_html(r.content)[1]
print(df)

Output:

       #             Name      Type  Category Contest  PP Power Accuracy   Gen
0      1            Pound    Normal  Physical   Tough  35    40     100%     I
1      2     Karate Chop*  Fighting  Physical   Tough  25    50     100%     I
2      3      Double Slap    Normal  Physical    Cute  10    15      85%     I
3      4      Comet Punch    Normal  Physical   Tough  15    18      85%     I
4      5       Mega Punch    Normal  Physical   Tough  20    80      85%     I
..   ...              ...       ...       ...     ...  ..   ...      ...   ...
821  822      Fiery Wrath      Dark   Special     ???  10    90     100%  VIII
822  823  Thunderous Kick  Fighting  Physical     ???  10    90     100%  VIII
823  824    Glacial Lance       Ice  Physical     ???   5   130     100%  VIII
824  825   Astral Barrage     Ghost   Special     ???   5   120     100%  VIII
825  826      Eerie Spell   Psychic   Special     ???   5    80     100%  VIII

[826 rows x 9 columns]

You can also send these values to a neat csv file by adding this line to your code:

df.to_csv('Moves.csv', index = False)

Screenshot of csv file:

How to improve Beautiful soup webscrape loop?

1 Answers