I'd like to watch a private website that displays numbers every s seconds, where s is a float and belongs to a range [2,5]. Also, there is not any API to do that.
I'm using Selenium + Python + JS. In short, I have:
Driver.py
~ a lot of code above ~
def __setOptions(self) :
logging.info("Setting ChromeOptions up.")
options = webdriver.ChromeOptions()
options.add_argument("start-maximized")
options.add_argument("--headless")
return options
def __desiredCapabilities(self) :
logging.info("Setting DesiredCapabilities up.")
desired_capabilities = DesiredCapabilities.CHROME
desired_capabilities['goog:loggingPrefs'] = { 'browser':'ALL' }
return desired_capabilities
self.__driver = webdriver.Chrome("../drivers/chromedriver",
options = self.__setOptions(),
desired_capabilities= self.__desiredCapabilities()
)
~ a lot of code below ~
Java Script:
~ a lot of code above ~
self._driver.execute_script("""
const targetNode = document.getElementById('lastvalueadded');
const config = { attributes: true, childList: true, subtree: true };
const callback = (mutationList, observer) => {
var IWANNATHISVALUE = "";
for (const mutation of mutationList) {
if (mutation.type === 'childList') {
newValue = document.querySelectorAll('div.history > div')[0].childNodes[0].nodeValue;
}
}
console.log("IWANNATHISVALUE: " + newValue);
};
try {
const observer = new MutationObserver(callback);
observer.observe(targetNode, config);
}catch (e) {
observer.disconnect();
}
""")
~ a lot of code below ~
Python:
~ a lot of code above ~
while 1:
for entry in self._driver.get_log('browser'):
if entry['level'] == 'INFO' and 'IWANNATHISVALUE' in entry['message'] :
IWANNATHISVALUE = entry['message'].split()[-1].replace('"','')
self.__dataProcess(IWANNATHISVALUE)
~ a lot of code below ~
In short, I use console.log(...) and get_log('browser') methods to display and catch, respectively, new values as soon as the website displays them (not by far the best solution). However, these values are displayed inside an iFrame object. For some unknown reason, I can not see INFO, DEBUG, or ERROR messages in my terminal (which includes my message "IWANNATHISVALUE: " + newValue), but WARNING and SEVERE messages. In contrast, I see all types of messages (INFO, DEBUG, ERROR, WARN, ...) in my chromedriver/browser console. Why that? I have tested it with and without --headless option.
For a test, I did not switch to any iFrame and executed console.log("test") and could see the message in my terminal. I concluded that INFO, DEBUG, and ERROR messages could not be shown in the terminal when we switched to an iFrame, only in the browser console.
My next step is to use Python Flask (to listen) and Ajax (to send) to pass information when MutationObserver detects a new value. Is there a better way to do that?
I do not have to use Selenium + Python + JS. I just want to catch these values. I am open to suggestions for new solutions.