Accessing an element text behind a tooltip selenium python

Question

Context: I (as a soil researcher) have been working on a script to automate the process of acquiring data from a large database site in the Netherlands.

The site is called https://www.dinoloket.nl/ondergrondmodellen

whenever you select a location you can get an expected soil build-up for different models. the model that I use is the regis v2.2. that shows different lithologie formations.

so far i have been able to create a script that selects the correct model, inputs a address and export the depths of the different layers

Example of what the site gives as layer output

in the image it also shows a tooltip that is different for each color in the column. I would like to be able to access this text however, it keeps disappearing.

I have read other related questions however, what is different is that the tooltip text is variable depending on where you are in the column.

tooltip on the site

in the image you can see that the tooltip (id="columns-tooltip") has a changing message whenever the location on the tooltip changes.

Does anybody has an idea on how to access the text "Complexe eenheid, bestaande uit een afwisseling van zandige klei, midden en fijn zand, klei en veen en een weinig grof zand"

I am aware that the question is vague, however some pointers in the right direction might already be helpful.

Thank you

[![driver = webdriver.Chrome(PATH)
driver.get("https://dinoloket.nl\ondergrondmodellen")

#---------------- % to the website
time.sleep(1)
#---------------- % access the frame where specific buttons are located

frame = driver.find_element_by_css_selector('#block-main-content > article > div > div > div > div > div.ondergrond-modellen-tab.active > div > div > div > iframe')
driver.switch_to.frame(frame)

#----------------
time.sleep(1)

#---------------- % select model by clicking the radiobutton next to it

selectmodel= driver.find_element_by_css_selector("#mapDiv > div.esri-view-user-storage > div > dinoloket-filter-toolbar > div > div > div:nth-child(4) > p-radiobutton > div > div.ui-radiobutton-box.ui-widget.ui-state-default")
selectmodel.click()


#--------- % find the magnifying glass button and input an address

confirmdialog = driver.find_element_by_id("idSearch-button")
confirmdialog.click()

input_address= driver.find_element_by_css_selector('#search-field > div > div > input')
input_address.send_keys("Rederijweg 26, Oosterhout")


#----------------
time.sleep(1)
#---------------- % hit enter

input_address.send_keys(Keys.RETURN)

#----------------
time.sleep(2)
#---------------- %find the drillcore button that gives access to the model

drillbutton = driver.find_element_by_id("drill-button")
drillbutton.click()

#----------------
time.sleep(1)
#---------------- % click in the map element to get the model of that specific point

start = driver.find_element_by_css_selector('body > dinoloket-app > div')
start.click()][1]][1]

user_3485434 · Answer

In order to make tool-tip presented you have to hover with mouse on some element.
Let's call that element hoverable.
If so your code can be something like this:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains

wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)

hoverable = driver.find_element_by_css_selector('the_element_locator')
actions.move_to_element(hoverable).perform()
tool_tip = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.mdl-columns-tooltip")))
tool_tip_text = tool_tip.text

Pay attention that tool_tip_text will contain all the text inside the tooltip, containing `Lithologie:' in this case.

user_9444073 · Answer

so basically for hovering in Selenium-Python, we use ActionChains.

from selenium.webdriver.common.action_chains import ActionChains
action = ActionChains(driver)

you can use move_to_element like below, to hover to element which has tool tip :-

action.move_to_element(driver.find_element_by_xpath('xpath of element which has tool tip')).perform()
tool_tip_text = driver.find_element_by_xpath("//div[@id='columns-tooltip']").text
print(tool_tip_text)

Now, there could be a situation and by reading your post, it seems like we may have more than one tool tip to extract, if so you can utilize find_elements instead of find_element, but make sure to pass a common locator if that happens to be.

user_13841248 · Answer

As an alternative, are you aware there's an API for your page. You can use that to harvest all the data you need at once for all elements without going near a browser. It will be significantly quicker and more stable.

Try this code:

import requests

url = 'https://www.dinoloket.nl/javascriptmodelviewer-web/rest/models/columns/descripted'

postData = {
    "language":"nl",
    "modelType":"RGS",
    "model":"REGIS",
    "depthReference":"NAP",
    "version":"v02r2",
    "dinoId":"B32F0034",
    "topDepth":"null",
    "bottomDepth":"null"
    }

response = requests.post(url=url, json=postData)
json_response = response.json()


for columns in json_response['columns']:
    for values in columns['profileMetadata']:
        for layer in values['layerInfos']:
            if layer['code'] == 'LITHOLOGY':
                print ("type: " , columns['columnType'])
                print ('upper: ', values['upper'])
                print ('lower' , values['lower'])
                print ('Code: ' , layer['code'])
                print ('value: ', layer['value'])
                print ('
###############
') # seperator

In the postData you can see "model":"REGIS" - which you say you're using.

The "dinoId":"B32F0034" seems to relate to the identifier:

When i run that code, i get the following output - which seems to be every "lithologie" tool tip on that page:

type:  HYDROGEOLOGY
upper:  344
lower 386
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit midden en grof zand, met weinig zandige klei, 
fijn zand en grind en een spoor klei en veen

###############

type:  HYDROGEOLOGY
upper:  21
lower 138
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit midden en fijn zand, weinig zandige klei en grof zand en een spoor klei, veen en grind

###############

type:  HYDROGEOLOGY
upper:  140
lower 197
Code:  LITHOLOGY
value:  Kleiige eenheid, hoofdzakelijk bestaande uit zandige klei en klei, weinig fijn en midden zand en een spoor veen en grof zand

###############

type:  HYDROGEOLOGY
upper:  199
lower 266
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit midden, grof en fijn zand, weinig kleiig zand 
en een spoor klei en grind

###############

type:  HYDROGEOLOGY
upper:  268
lower 335
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit grof en midden zand, weinig zandige klei, fijn zand en grind en een spoor klei

###############

type:  HYDROGEOLOGY
upper:  337
lower 342
Code:  LITHOLOGY
value:  Kleiige eenheid, hoofdzakelijk bestaande uit zandige klei weinig klei, fijn, midden en grof zand, een spoor grind en een kans op stenen, keien en blokken

###############

I have no idea if lower and upper are of any use to you - they're just there as sample data for the extract.

Depending on what other infromation you need, the JSON data structure you get back from the site can be explored in dev tools on the network tab. Have this tab open in the background when you use your site and find the item named descripted to browse

* * * * * update * * * * * *

Chaining together the APIs to allow for a drill at address.

Update the address at the start of the script and it will output the rest. Over to you to get the output format you want:

import requests
import urllib.parse

#Enter the address to drill - this is all that's needed
address = urllib.parse.quote("Rederijweg 26, Oosterhout")


#user the search suggested to get the ID for the address 
suggest_response = requests.get('https://geodata.nationaalgeoregister.nl/locatieserver/v3/suggest?q='+address)
suggest_json = suggest_response.json()
address_ID = suggest_json['response']['docs'][0]['id']
#address id  is something like" adr-21ceef9f7c9e361352969a3bcab636e6"


#use the address ID to get the coordinates:
lookup_response = requests.get("https://geodata.nationaalgeoregister.nl/locatieserver/v3/lookup?fl=centroide_rd,type&id=" + address_ID)
lookup_json = lookup_response.json()
coords_string =lookup_json['response']['docs'][0]['centroide_rd']
#coords come in like: POINT(117454.426 408494.3)
# prcess this into 2 values
coords_string= str.replace(coords_string, "POINT(", "")
coords_string= str.replace(coords_string, ")", "")
coords_array = str.split(coords_string, " ")


#use those coordinates to drill
url = "https://www.dinoloket.nl/javascriptmodelviewer-web/rest/models/columns/virtual"
postData = {
  "language": "nl",
  "modelType": "RGS",
  "model": "REGIS",
  "depthReference": "MV",
  "version": "v02r2",
  "resolution": "100",
  "ycoordinate":float(coords_array[1]),
  "xcoordinate":float(coords_array[0])
}

response = requests.post(url=url, json=postData)
json_response = response.json()

#process the drill results
for columns in json_response['columns']:
    for meta in columns['profileMetadata']:
        for layer in meta['layerInfos']:
            if layer['code'] == 'LITHOLOGY':
                print ("type: " , columns['columnType'])
                print ('upper: ', meta['upper'])
                print ('lower' , meta['lower'])
                print ('Code: ' , layer['code'])
                print ('value: ', layer['value'])
                print ('
###############
') # seperator

For me, TOP 3 outputs only (save some paste-space):

type:  HYDROGEOLOGY
upper:  20
lower 21
Code:  LITHOLOGY
value:  Complexe eenheid, bestaande uit een afwisseling van zandige klei, midden en fijn zand, klei en veen en een weinig grof zand

###############

type:  HYDROGEOLOGY
upper:  21
lower 22
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit midden en fijn zand, met weinig zandige klei en grof zand en een spoor klei, veen en grind

###############

type:  HYDROGEOLOGY
upper:  22
lower 23
Code:  LITHOLOGY
value:  Zandige eenheid, hoofdzakelijk bestaande uit midden en fijn zand, met weinig zandige klei en grof zand en een spoor klei, veen en grind

No idea how unique these descriptions are across the site - but it matches up:

Accessing an element text behind a tooltip selenium python

3 Answers

* * * * * **update** * * * * * *

* * * * * update * * * * * *