BeautifulSoup scraping .text attribute problem

user_14227118 0
Viewed 56

I have the following code to scrape a page, https://www.hotukdeals.com

from bs4 import BeautifulSoup
import requests

url="https://www.hotukdeals.com/hot"
r = requests.get(url)
soup = BeautifulSoup(r.text,"html.parser")
deals = soup.find_all("article")
for deal in deals:
    priceElement = deal.find("span",{"class":"thread-price"})
    try:
        print(priceElement,priceElement.text)
    except AttributeError:
        pass

For some reason, this works, scraping the prices of deals, for a certain number of times in the loop, and then stops working.

Program output:

<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£9.09</span> £9.09
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£39.95</span> £39.95
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£424.98</span> £424.98
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£8.10</span> £8.10
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£14.59</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£2.50</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl text--color-greyShade">£20</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£19</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£29</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl text--color-greyShade">£49.97</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">FREE</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£2.49</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£1.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£54.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£12.85</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£1.99</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£21.03</span>
<span class="thread-price text--b cept-tp size--all-l size--fromW3-xl">£5.29</span>

As you can see from the output, after the first four lines, the .text attribute is empty, but there is text in the element.

Anybody know anything about this? Any ideas or solutions?

web-scraping beautifulsoup html python
1 Answers

Beautifulsoup needs html5lib parser to parse the site correctly, e.g.:

import requests
from bs4 import BeautifulSoup

url = "https://www.hotukdeals.com/"

soup = BeautifulSoup(requests.get(url).content, "html5lib")  # <-- use html5lib

for price in soup.select(".thread-price"):
    print(price.text)

Prints:

£149.99
£7
£21.03
£31.79
£359.10
£19.99
£60
£0.60
£168
£4.99
£20
£119
Free P&P
Free
£5
£89.99
FREE
£10.96
£1.79
user_100359850
Related