IndexError while iterating

user_15132317 0
Viewed 51

I have a problem with occurring IndexError while iterating. The program works well until everything is done, there are no more "sub websites" to go to and then it crashes and because of that, it is impossible to save in .txt.

Traceback (most recent call last)

newUrl = nextpage[counter]['href']
IndexError: list index out of range

Code

from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
import json
class Olx():

    def __init__(self, url):
        self.url = url

    def getPrice(self):
        """Get prices from olx"""
        html = urlopen(self.url)
        bs = BeautifulSoup(html, 'html.parser')
        price = bs.findAll('p', class_='price')
        return price

    def nextPage(self):
        """Go to the next page"""
        html = urlopen(self.url)
        bs = BeautifulSoup(html, 'html.parser')
        pageButton = bs.findAll('a', {'class': 'block br3 brc8 large tdnone lheight24'})
        try:
            return pageButton
        except AttributeError:
            None
        else:
            return pageButton

    

olxprices = Olx('https://www.olx.pl/nieruchomosci/mieszkania/wynajem/olsztyn/').getPrice()
nextpage = Olx('https://www.olx.pl/nieruchomosci/mieszkania/wynajem/olsztyn/').nextPage()
counter = 0

output = []
while len(nextpage) > 0:
    for price in olxprices:
        output.append(price.get_text().strip())
        print(price.get_text().strip())
    newUrl = nextpage[counter]['href']
    olxprices = Olx(newUrl).getPrice()
    counter += 1

print(output)
index-error beautifulsoup iteration python loops
2 Answers

len(nextpage) never changes, so the while loop never ends and eventually counter indexes past the end of nextpage. Instead, do something like:

for page in nextpage:
    for price in olxprices:
        output.append(price.get_text().strip())
        print(price.get_text().strip())
    newUrl = page['href']
    olxprices = Olx(newUrl).getPrice()
user_2356980

You could try using an exception.

while len(nextpage) > 0:
    try:
        for price in olxprices:
            output.append(price.get_text().strip())
            print(price.get_text().strip())
        newUrl = nextpage[counter]['href']
        olxprices = Olx(newUrl).getPrice()
        counter += 1
    except IndexError:
        break

(or do whatever you want as an exception there) if that doesnt answer your question, maybe its because the length of the page stays stationary so you might want to iterate through that as well.

user_148409600
Related