I'm working on a project that involves scraping some product data off of a vendor's web site (with their blessing, but not their help). I'm working in a C# shop, so I'm using the .NET Windows Forms WebBrowser control.
I'm responding to the document completed event, but I'm finding that I have to thread sleep for a little bit, or else the data doesn't show up where I expect it to in the DOM.
In looking at the javascript on the page, I can see that it is dynamically altering the existing DOM content (setting someDomElement.innerHTML) after the page finishes loading. It's not making any ajax calls, it's using data it already has from the original page load. (I could try and parse for that data, but it is embedded in javascript and it's a bit obfuscated.) So evidently I'm somehow getting the document completed event Before the javascript has finished running.
There could eventually be a lot of pages to scrape, so waiting around for a half second or whatever is really far less than ideal. I would like to only wait until all the JavaScript that starts on document ready / page load has finished running before I examine the page. Does anyone know of a way to do that?
I suppose the document completed event shouldn't fire until then, right? But it definitely appears to be. Maybe somewhere the page javascript is using a setTimeout. Is there a way to tell if there any timeouts pending?
Thanks for any help!