if statement not working with scraped web text

user_15763204 0
Viewed 52

I understand this is a fairly common question, but the circumstances of this make it a perplexing one.

I am using beautifulsoup to scrape certain data from a website, using this I am also checking for a "Next page" link on the page I am currently scraping to see if I can scrape another.

next_page_button_finder = soup.find('ul', class_='navnext').text

To check how this comes out, I print it using:

print(next_page_button_finder)

The output is:

Next >>

However, and this is the curious part, when I then try to validate this by creating the simple if statement:

    if next_page_button_finder == "Next >>":
        print("yes")
    else:
        print("no")

"no" is printed.

Any help would be hugely appreciated.

Here is the code you can use to replicate the issue (any link from spareroom.com will work, however, for your ease you can use this link https://www.spareroom.co.uk/flatshare/?search_id=1034984872& ) :

from bs4 import BeautifulSoup
import requests

html_address = input("Paste page the address here:")

html_text = requests.get(html_address).text

soup = BeautifulSoup(html_text, 'lxml')
prices = soup.find_all('strong', class_='listingPrice')

next_page_button_finder = soup.find('ul', class_='navnext').text

print(next_page_button_finder)

if next_page_button_finder == "Next >>":
    print("yes")
else:
    print("no")
if-statement beautifulsoup python string
4 Answers

What debugging shows

If you debug and stop at the if statement, you will see that the text actually contains newline characters before and after. You could use that string, or simply strip your string beforehand with string.strip()

user_120332710

A better way might be to simply remove that potential for variation in text and simply test whether the "button" is present in the html using its id (of the actual element not its parent - which you are currently targeting). The id will also allow for faster matching and this test should be more reliable.

import requests
from bs4 import BeautifulSoup as bs
  
links = ['https://www.spareroom.co.uk/flatshare/?search_id=1034984872&',
         'https://www.spareroom.co.uk/flatshare/?offset=10&search_id=1034984872&sort_by=age&mode=list',
         'https://www.spareroom.co.uk/flatshare/?offset=410&search_id=1034984872&sort_by=age&mode=list']

with requests.Session() as s:
    for link in links:
        r = s.get(link)
        soup = bs(r.content, 'lxml')
        next_page = soup.select_one('#paginationNextPageLink')
        if not next_page is None:
            print('Yes')
        else:
            print('No. Last page.')
user_62412350

I think the problem is from a trailing whitespace. In other words, there is a space character at the start, or in this case, end, of the string. Due to this space, the strings are not the same, and when compared, will result in a False statement (since they are not equal).

The string that you are comparing the variable to is "Next >>", however the string you said the variable had stored is "Next >> " (There is a space at the end of this string). Therefore, they are not the same and will result in False.

How do you fix this problem? If you use the .strip() method, then it removes whitespace around the string. If you update your code to the following, it will work:

if next_page_button_finder.strip() == "Next >>":
    print("yes")
else:
    print("no")
user_119857430

Using:

print(list(next_page_button_finder))

I was able to see that the "next_page_button_finder" str actually had newline characters before and after the text:

['\n', 'N', 'e', 'x', 't', ' ', '>', '>', '\n']

And so I changed the if statement to:

if next_page_button_finder == "\nNext >>\n":
    print("yes")
else:
    print("no")

And now it prints:

yes

Alternatively, I could write one line of code:

next_page_button_finder = next_page_button_finder.strip()

To strip away the newline characters and the original code would work just fine.

user_157632040
Related