extract text from h1 and id with python beautiful soup

Viewed 483

I'm trying to extract the text from HTML id="itemSummaryPrice" but I couldn't figure it out.

html = """
<div id="itemSummaryContainer" class="content">
            <div id="itemSummaryMainWrapper">
                <div id="itemSummaryImage">
                    <img src="https://img.rl.insider.gg/itemPics/large/endo.fgreen.891c.jpg" alt="Forest Green Endo">
                </div>
                <h2 id="itemSummaryTitle">Item Report</h2>
                <h2 id="itemSummaryDivider"> | </h2>
                <h2 id="itemSummaryDate">Friday, January 15, 2021, 8:38 AM EST</h2>
                <div id="itemSummaryBlankSpace"></div>
                <h1 id="itemSummaryName">
                    <span id="itemNameSpan" style="color: rgb(88, 181, 73);"><span>Forest Green</span> <span>Endo</span></span>
                </h1>
                **<h1 id="itemSummaryPrice" style="color: rgb(88, 181, 73);">200 - 300</h1>**
            </div>
        </div>
"""

my code:

price_checker_site = requests.get(price_checker_url + match2)
price_checker_site_soup = BeautifulSoup(price_checker_site, 'html.parser')
price_check_item = price_checker_site_soup.find('h1', {'id': 'itemSummaryPrice'})

print(price_check_item)

returns with:

<h1 id="itemSummaryPrice"></h1>

What I'm trying to extract:

<h1 id="itemSummaryPrice">200 - 300</h1>
OR
<h1 id="itemSummaryPrice" style="color: rgb(88, 181, 73);">200 - 300</h1>
OR
200 - 300
2 Answers

Because I can't place comments yet an answer then. Shouldn't you call .text behind the price_check_item?

So the python code looks like this.

price_checker_site = requests.get(price_checker_url + match2)
price_checker_site_soup = BeautifulSoup(price_checker_site, 'html.parser')
price_check_item = price_checker_site_soup.find('h1', {'id': 'itemSummaryPrice'})

print(price_check_item.text) #Also possible to do print(price_check_item.text.strip())

I think this is the correct answer. Unfortunately not able to test now. Will check my code for you tonight.

As discussed in the comments, the content you seek is loaded dynamically using JavaScript. Therefore, you must either use a library like Selenium to dynamically run the JS, or find out where/how the data is loaded and replicate that.

Method 1: Use Selenium

from selenium import webdriver

url = 'https://rl.insider.gg/en/psn/octane/grey'
driver = webdriver.Firefox(executable_path='YOUR PATH')  # or Chrome
driver.get(url)
price = driver.find_element_by_id('itemSummaryPrice')
print(price.text)

In this case its easy, you just make the request and use find_element_by_id to get the data you want.

Method 2: Trace & Replicate

If you look at your browser's debugger, you can find where/how the itemSummaryPrice it set.

In particular, we find that its set using $('#itemSummaryPrice').text(itemData.currentPriceRange) in https://rl.insider.gg/js/itemDetails.js.

The next step is to find out where itemData comes from. It turns out, this is not from some other file or API call. Instead, it appears to be hard-coded in the HTML source itself (presumably loaded server-side).

If you inspect the source, you'll find the itemData is just a JSON object defined on one line within a script tag on the page itself.

There are two different approaches you can use here.

Use Selenium's execute_script to extract the data. This gives you the JSON object in a ready-to-use format. You can then just index it to get the currentPriceRange.

from selenium import webdriver

driver = webdriver.Firefox(executable_path='YOUR PATH')  # or Chrome
driver.get(url)
itemData = driver.execute_script('return itemData')
print(itemData['currentPriceRange'])

Method 2.1: Alternative to Selenium

Alternatively, you can extract this in Python using traditional methods. Then, convert that to a usable Python object using json.loads and then, index the object to extract the currentPriceRange -- this gives you the desired output.

import re
import requests
import json

# Download & convert the response content to a list
url = 'https://rl.insider.gg/en/psn/octane/grey'
site = str(requests.get(url).content).split('\\n')

# Extract the line containing 'var itemData'
itemData = [s for s in site if re.match(r'^\s*var itemData', s)][0].strip()

# Remove 'var itemData' and ';' from that line
# This leaves valid JSON which can be converted from a string using json.loads
itemData = json.loads(re.sub(r'var itemData = |;', '', itemData))

# Index the data to extract the 'currentPriceRange'
print(itemData['currentPriceRange'])

This approach doesn't require Selenium to run the JavaScript and also doesn't require BeautifulSoup to parse the HTML. It does rely on the itemData being initialized in a certain way. Should the developers of that site decide to change the way this is done, you'll have to adapt it slightly in response.

Which method should I use?

If all you really want is the price range and nothing else, then use the first method. If you're interested in other data as well, you'd be better off extracting the full itemData JSON from the source and using that.

One could argue Selenium is more reliable than manual parsing of the HTML, but in this case you're probably fine. In both cases, you assume there is some itemData defined somewhere. If the format does change slightly, then the parsing may break. The other disadvantage is if part of the data relied on JS function calls -- which Selenium would execute, whereas manual parsing couldn't account for. (This isn't the case here, but it could change).

Related