I am trying to scrape data from a table of sporting statistics presented as HTML using the BeautifulSoup and requests libraries. I am running both of them on Python 3.5. I seem to be successfully obtaining the HTML via requests because when I display r.content, the full HTML of the website I am trying to scrape is displayed. However, when I pass this to BeautifulSoup, BeautifulSoup drops the bulk of the HTML which are the tables of statistics themselves.
If you take a look at the website in question, the HTML from "Scoring Progression" onward is dropped.
I think the problem relates to the pieces of HTML which are included between brackets ('[' and ']') but I have not been able to develop a workaround. I have tried the html, lxml and html5lib parsers for BeautifulSoup, to no avail. I have also tried providing 'User-Agent' headers and that did not work either.
My code is as below. For brevity's sake I have not included the output.
import requests
from bs4 import BeautifulSoup
r = requests.get('http://afltables.com/afl/stats/games/2015/031420150402.html')
soup = BeautifulSoup(r.content, 'html5lib')
print(soup)