Python selenium - get values from p and span

Question

Let's say I have a code like this:


            
            X:  ValueX
            A:  ValueA  
            B: ValueB
            C: ValueC

I would like to get a dictionary like {X:ValueX, A:ValueA, B:ValueB, C:ValueC}

I tried to get driver.find_elements_by_xpath('//div[@class = "div1") but what next?

How do I go deeper in the structure? How do I retrieve values of span nested in p ?

user_10784315 · Answer

Copy the xpath of each of the value you want to retrieve. You will find a pattern there and a simple loop can be used to iterate those elements.

For example, the xpaths of elements inside the tag might look like these:

- /html/body/div/div/p[1]/span
- /html/body/div/div/p[2]/span
- /html/body/div/div/p[3]/span

Following this pattern, you can use the following loop to extract your desired value:

for index in range(1,4):

    # concatenating the index values to iterate through the elements
    xpath_string = "/html/body/div/div/p[" + str(index) + "]/span"
  
    # catching that particular element using xpath
    element = driver.find_element_by_xpath(xpath_string)

user_3485434 · Answer

In XML your code I see the first text pair is inside

while all the other text pairs are inside

elements. So the common denominator here is that all these elements are immediate children of

element.
Now you can get the text value of the direct child elements and of the internal span elements.
You can get a list of "key" values, a list of "value" values and then zip them into a dictionary as following:

keys = []
values = []
elements = driver.find_elements_by_xpath("//div[@class='div2']/*")
for element in elements:
    entire_text = element.text
    key = entire_text.split(" ")[0]
    keys.append(key)
elements = driver.find_elements_by_xpath("//div[@class='div2']/*/span")
for element in elements:
    value = element.text
    values.append(value)
dictionary = dict(zip(keys, values))
print(dictionary) #{X:ValueX, A:ValueA, B:ValueB, C:ValueC}

user_5953932 · Answer

Use beautifulsoup! It allows you to search for specific tags/ids/classes and then extract the text/links etc.

Here's the docs: https://beautiful-soup-4.readthedocs.io/en/latest/

Python selenium - get values from p and span

3 Answers