The goal is to download a file from a website using HtmlUnit. The download link is generated in JavaScript and dynamically injected into a dropdown menu, based on a previous selection in another dropdown menu. Both dropdown menus are dynamically injected via JavaScript and have the same element structure.
So in general, when loading the page, it shows only the first dropdown list. This sample shows the initial element structure when the page is loaded. When selecting an item from that list, the second dropdown list appears and the element's source structure has been updated to this. When clicking the second dropdown list, the page shows a list with several options containing a direct link to a file. This example shows the selection of the first dropdown list is made, showing the options of the second dropdown list (with a green download icon). At this moment, the source has been updated to with this code (i.e. similar to the previous div). Clicking on any of the list items will start an immediate download of the required file.
What happens in my HtmlUnit application is, when I trigger the same steps until the second dropdown list should appear, the div with id 'formatSelect' stays empty. It looks like the JavaScript code is not being executed for that part only, while it does so for the first part.
A brief sample of my code:
private static final String URL = "https://some.url.org/";
private static final String xpathDeltaSelect = "div[@id='div1']";
private static final String xpathFormatSelect = "div[@id='div2']";
private static final String xpathDropDownToggle = "div[@class='dropdown-toggle']";
private static final String xpathSearchButton = "div/input[@type='search']";
private static final String xpathListBox = "div/ul[@role='listbox']";
webClient_ = new WebClient(BrowserVersion.CHROME);
webClient_.getOptions().setRedirectEnabled(false);
webClient_.getOptions().setDownloadImages(false);
webClient_.getOptions().setThrowExceptionOnScriptError(false);
webClient_.getOptions().setThrowExceptionOnFailingStatusCode(false);
HtmlPage page = webClient_.getPage(URL);
webClient_.waitForBackgroundJavaScript(JAVASCRIPT_TIMEOUT);
final HtmlDivision divDeltaSelect = selectWrapper(page).getFirstByXPath(xpathDeltaSelect );
final HtmlDivision divDeltaDropdown = divDeltaSelect.getFirstByXPath(xpathDropDownToggle);
final HtmlInput inputDelta = divDeltaDropdown.getFirstByXPath(xpathSearchButton);
inputDelta.focus();
final HtmlUnorderedList ulDeltaSelect = divDeltaSelect.getFirstByXPath(xpathListBox);
Iterator<DomElement> iterator = ulDeltaSelect.getChildElements().iterator();
// Iterate to the desired item and 'click'
...
page = li.click();
break;
}
// So far, everything goes well.
final HtmlDivision divFormatSelect = selectWrapper(page).getFirstByXPath(xpathFormatSelect);
System.out.println(divFormatSelect.asXml());
// At this point, divFormatSelect is still empty...
The output of the sysout at the end is the following:
<div data-v-0d6d77a2="" id="formatSelect">
<!---->
</div>
The selectWrapper method is briefly doing the following (because there are 3 similar "selectWrapper" divs that return different data files, but I only need the first one):
final List<DomElement> elements = page.getElementsById("selectWrapper");
...
return elements.get(0);
I did notice that the initial page load at line
HtmlPage page = webClient_.getPage(URL);
throws the following JavaScript error:
com.gargoylesoftware.htmlunit.ScriptException: missing ) after formal parameters (https://some.url.org/path/to/js/javascript.js#1)
... + stacktrace
Could this be related to this bug? However, I do not understand why the first injected code can be retrieved, but not the second part, while the browser interface does the processing correctly. Am I missing something?