Webscraping CME

user_3697498 0
Viewed 642

I am trying to webscrape the following:

https://www.cmegroup.com/trading/interest-rates/us-treasury/10-year-us-treasury-note_quotes_volume_voi.html#tradeDate=20210507

In particular to get Totals in that First table (block trades, EFP,EFR etc)

When I inspect the page I get different results than when I actually go to scrape and get the "page Source" data. This is making it difficult to find the data for me (im new to this)

After some exploring I found the data in https://www.cmegroup.com/CmeWS/exp/voiProductDetailsViewExport.ctl?media=json&tradeDate=20210507&reportType=F&productId=316 which is an excel file

My code so far is

header= {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) ' 
  'AppleWebKit/537.11 (KHTML, like Gecko) '
  'Chrome/23.0.1271.64 Safari/537.11',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.3',
  'Accept-Encoding': 'none',
  'Accept-Language': 'en-US,en;q=0.8',
  'Connection': 'keep-alive'}

url = "https://www.cmegroup.com/content/cmegroup/en/trading/interest-rates/us-treasury/10-year-us-treasury-note_quotes_volume_voi.html"
r = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
response = urlopen(r)
soup = BeautifulSoup(response, 'lxml')

In short does anyone have a recommendation on a better way to do this than having to sort through Excel files? Thanks!

web-scraping beautifulsoup python
1 Answers

When visiting this page in a browser, the DOM is populated asynchronously using JavaScript. You can expect BeautifulSoup not to work for pages like this, since BeautifulSoup can only see content which is baked directly into the document at the time the server serves it to you.

Logging my network traffic reveals that the browser made several XHR (XmlHttpRequest) HTTP GET requests to some REST APIs. One of them returns JSON containing the information you're looking for. All you have to do is imitate that HTTP GET request:

def main():

    import requests

    url = "https://www.cmegroup.com/CmeWS/mvc/Volume/Details/F/316/20210507/F"

    params = {
        "tradeDate": "20210507",
        "pageSize": "50",
        "_": "1620683546888"
    }

    headers = {
        "Accept": "application/json",
        "Accept-Encoding": "gzip, deflate",
        "User-Agent": "Mozilla/5.0"
    }

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()

    data = response.json()

    print("Block trades (total volume): {}".format(data["totals"]["blockVolume"]))
    print("EFP (total volume): {}".format(data["totals"]["efpVol"]))
    print("EFR (total volume): {}".format(data["totals"]["efrVol"]))

    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Output:

Block trades (total volume): 7,500
EFP (total volume): 23,958
EFR (total volume): 34,486
>>>

Take a look at this other answer I posted, where I go more in-depth on how to log your network traffic, finding REST API endpoints and imitating requests.

user_109874320
Related