I have been making a project (mostly in python) which webscrapes a website for the results. Trouble is, the website loads critical information using AJAX after all the DOM elements are loaded, and the information loaded is hidden behind a maze of difficult-to-decode JS.
I am looking for a solution which loads (or runs) all the AJAX and gets the resulting HTML. This could be done with any coding language (since at this point, after days of searching, any solution is a solution) and could involve downloading the JS and HTML and running it locally, so long as no browser is opened in the process.
Example site:
index.html
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width">
<title>Example</title>
</head>
<body>
<div id="message">Hello World!</div>
<script src="/scripts/script.js"></script>
</body>
</html>
/scripts/script.js
window.onload = function loadDoc() {
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
document.getElementById("message").innerHTML =
this.responseText;
}
};
xhttp.open("GET", "ajax_info.txt", true);
xhttp.send();
}
ajax_info.txt
Goodbye World.