Further table url link scraping

A small adjustment to the try/except block, that writes the url and also the status of scraping:

    try:
        df = pd.read_html(url)[0]
        df['url'] = url
        df['scraped'] = True
        data.extend(df.to_dict(orient = 'records'))
    except:
        print(f'No tables found:{url}')
        data.append(dict(url=url, scraped=False))

Example

Limited to for url in urlList[5:10] to not iterate all of the links in this example, just kick the [5:10] to get all:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.rootsandrain.com/organiser21/uci/events/filters/dh/'
response = requests.get(url)
soup = BeautifulSoup(response.content)

urlList = ['https://www.rootsandrain.com'+row.a['href'] for row in soup.select('#T1 tbody tr')]

data = []

for url in urlList[5:10]:
    try:
        df = pd.read_html(url)[0]
        df['url'] = url
        df['scraped'] = True
        data.extend(df.to_dict(orient = 'records'))
    except:
        print(f'No tables found:{url}')
        data.append(dict(url=url, scraped=False))
        
pd.DataFrame(data)

#to get rid of all these NaN columns
#pd.DataFrame(data).dropna(axis=1, how='all')

Output