I am new to web crawl and only have some experience on crawling static sites. I have following questions for my new project:
Background
On this page: https://leetcode.com/contest/biweekly-contest-65/ranking/1/, a window that contains some code would pop up after you click on each file button.
Objective
My goal is to crawl code from all file buttons on this page and save them in local.
Challenge
I am not sure how to get code from all file buttons using web crawler.
What I have tried
I checked from inspection that by clicking the file button everytime, it generates a GET Request to api. For example, if you click the first (top left) button, it calls server with https://leetcode.com/api/submissions/586506999/ which contains the code I need in json format.
So looks like as long as i can gather all submission ids (586506999), i could receive all codes from this page, each in a json file.
However, I am not sure how to let the crawler get the subsmission id (586506999). I checked from the html looks the id isnt there, so a bit confused on how this id is generated and how to retrieve all of them on this page.
Also I am open to other strategies that can achieve my objective.
Thanks in advance