I have a question that seems to have been asked before, but is a bit different. I'm trying to scrape data from this website but the problem is that is seems like it's loaded with AJAX. Because of that my application is unable to find the id's and classes in the HTML that I'm looking for.
You can reproduce this by inspecting an element or viewing the source. Whilst viewing the source I'm seeing a lot less than whilst inspecting an element.
I thought that I could track down the file that contains the AJAX to load this html by pressing F12, going to the network tab and selecting XHR, but I'm unable to find it.
My question is: how do I retrieve this data or find out what file is used to collect the data?
An example of my code (I'm unable to find the Timetable_toolbar_elementSelect_popup0):
private async Task GetHtmlDocument(string url)
{
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
//request.Credentials = new LoginCredentials().Credentials;
try
{
WebResponse myResponse = await request.GetResponseAsync();
HtmlDocument htmlDoc = new HtmlDocument();
htmlDoc.OptionFixNestedTags = true;
htmlDoc.Load(myResponse.GetResponseStream());
var test = htmlDoc.GetElementbyId("Timetable_toolbar_elementSelect_popup0");
}
catch (Exception e)
{
}
}