I am looking for the smartest way to scrape a website with multiple subpages.
For example the page: https://www.fussballdaten.de/wettbewerbe/
Here we have many subpages for different leagues like: '/bundesliga/', '/regionalliga/', '/regionalliga/aufstiegsspiele/' etc.
As you can see here for example /regionalliga/ has many different subpages.
On the respective page /regionalliga/ i cant scrape anything cause there arent any necessary html tags. So the scraper has to continue with the next subpages.
What is the best way to scrape all these subpages?
Do I have to write them all in my defined leaguelist at the beginning? That would be an infinite list. I mean as an example something like this:
from bs4 import BeautifulSoup
import requests
import pandas as pd
leaguelist= ('/bundesliga/', '/regionalliga/', '/regionalliga/aufstiegsspiele/')
for x in leaguelist:
print(f'https://fussballdaten.de/{x}')
Thanks for all the help. I've been out of Python for a few years and would like to get back into it more intensively.