While loop for the entire list not just a single index lxml

Babiqowski 0
Viewed 59

Ideally I would like to keep my code as is and only supplement it with a while loop which will output all elements of the list until the final part of the list (I know there is probably - a simpler solution but I want to improve my knowledge of the while loop in context of my own example).

import requests
import lxml.html as lh
from lxml.etree import tostring
req=requests.get('https://www.dailymail.co.uk/debate/article-11113609/DAN-WOOTTON-intolerant-left-belittles-death-threats-against-JK-Rowling-peril.html#comments-11113609')
df=lh.fromstring(req.text)
### Can we use a while loop here to output all the results of the entire list iteration below
f=0+1
elem = df.xpath('//script')[f]
print(tostring(elem))

Example:

elem = df.xpath('//script')[0]
print(tostring(elem))

output:

b"\n var disableAds = true;\n PageCriteria = window.PageCriteria || {};\n PageCriteria.clientIP = '83.20.32.187';\n PageCriteria.nonAdservable = '' === 'true';\n PageCriteria.device = 'other';\n PageCriteria.liveCommentary = false;\n\n\n\n"

I am trying to get my code to output the entire result of the below parts of the list but using while loop instead of having to list all the list indexes manually.

elem = df.xpath('//script')[0]
elem = df.xpath('//script')[1]
elem = df.xpath('//script')[2]

until the last index of the list.

lxml xpath while-loop web-scraping python
1 Answers

Not sure if this is something you wish to achieve:

import requests
import lxml.html as lh
from lxml.etree import tostring

link = 'https://www.dailymail.co.uk/debate/article-11113609/DAN-WOOTTON-intolerant-left-belittles-death-threats-against-JK-Rowling-peril.html#comments-11113609'

req = requests.get(link)
df = lh.fromstring(req.text)
script_count = len(df.xpath('//script'))

for index in range(script_count):
    elem = df.xpath('//script')[index]
    print(tostring(elem))

Using while loop:

req = requests.get(link)
df = lh.fromstring(req.text)

index = 0
while True:
    try:
        elem = df.xpath('//script')[index]
    except IndexError:
        break

    print(tostring(elem))
    index+=1
asmitu0
Related