I am trying to scrape the contents of bidding websites, but am unable to fetch the complete page of the website . I am using crowbar on xulrunner to fetch the page first (as ajax loads certain elements in lazy fashion) and then scrape from the file. But on the mainpage of bidrivals website, this fails even when the local file is well formed. jSoup simply seems to end with '...' characters midway in the html code. If anyone has encountered this before, please help. The following Code is called for [this link].
File f = new File(projectLocation+logFile+"bidrivalsHome");
try {
f.createNewFile();
log.warn("Trying to fetch mainpage through a console.");
WinRedirect.redirect(projectLocation+"Curl.exe -s --data \"url="+website+"&delay="+timeDelay+"\" http://127.0.0.1:10000", projectLocation, logFile+"bidrivalsHome");
} catch (Exception e) {
e.printStackTrace();
log.warn("Error in fetching the nameList", e);
}
Document doc = new Document("");
try {
doc = Jsoup.parse(f, "UTF-8", website);
} catch (IOException e1) {
System.out.println("Error while parsing the document.");
e1.printStackTrace();
log.warn("Error in parsing homepage", e1);
}