Python Pandas Scraping Multiple Webpages to Multiple Data Frames?

user_18003175 0
Viewed 31

Noob here. I am trying to organize traits per token number, luckily this webpage has them listed starting from /0. https://ape.offbase.org/token/0

I cant seem to get them into a usable dictionary to work with. I need to be able to lookup the token number and reference its traits. Tokens also have different trait counts. This is what I have so far:

URL = 'https://ape.offbase.org/token/'


for page in range(0,10):


req = requests.get(URL + str(page) + '/')
dfs = pd.read_html(URL + str(page) + '/')
df = dfs[0:10]
print(df)

Results:

[            0            1
0  Background       Orange
1     Clothes  Striped Tee
2     Earring  Silver Hoop
3        Eyes       X Eyes
4         Fur        Robot
5       Mouth   Discomfort]
web-scraping dataframe dictionary python
1 Answers

You could try defining a dictionary.

Where each key will be the loop iterator and the value will be a DataFrame, as suggested in the comment

URL = 'https://ape.offbase.org/token/'

results = {}
for page in range(0,10):

    req = requests.get(URL + str(page) + '/')
    value = pd.read_html(URL + str(page) + '/')[0]
    
    results[page] = value

would work as a pseudo-list

result[0]
0 1
Background Orange
Clothes Striped Tee
Earring Silver Hoop
Eyes X Eyes
Fur Robot
Mouth Discomfort
user_115390760
Related