Web scraping with bs4 does not return number value

user_16068550 0
Viewed 88

The page I'm trying to scrape from is https://www.investagrams.com/Stock/ac and I'm trying to get the price value (779.00) but my code only returns: {{ViewStockPage.Data.Stock.LatestStockHistory.Last | numberPriceFormat}}

My code:

from bs4 import BeautifulSoup
import requests

r = requests.get('https://www.investagrams.com/Stock/ac')
soup = BeautifulSoup(r.text, "lxml")

main = soup.find('div', class_= 'd-flex flex-row justify-content-between')

header = main.find('h4', class_= 'mb-0')

price = header.find('span', class_= 'mr-2').string

print(price)

Website HTML:

<h4 class="mb-0"> 
<small class="ng-binding">Ayala Corporation (PSE:AC) </small> <br>
<strong> 
 <span class="mr-2 ng-binding" data-ng-class="ViewStockPage.Data.Stock.LatestStockHistory.LastClass">779.00 </span> 
 <span data-ng-class="{'stock-up-caret stockprice-up' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage > 0, 'stock-down-caret stockprice-down' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage < 0, 'glyphicon glyphicon-minus stockprice-flat': ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage == 0}" class="stock-down-caret stockprice-down" style=""> </span> 
 <span style="font-size: 13px; vertical-align: middle;" data-ng-class="{'stockprice-up' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage > 0, 'stockprice-down' : ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage < 0, 'stockprice-flat': ViewStockPage.Data.Stock.LatestStockHistory.ChangePercentage == 0}" class="stockprice-down"> 
   <span class="ng-binding">-21.00 </span> 
   <span class="ml-1 ng-binding">-2.62% </span> 
 </span> 
</strong> 
</h4>
web-scraping beautifulsoup python
2 Answers

This page uses JavaScript to add value in place {{...}} but requests and Beautifulsoup can't run JavaScript. You may need Selenium to control real web browser which can run JavaScript.

Using DevTools in Firefox/Chrome (tab: Network, filter: XHR) I found that JavaScript reads data from

https://webapi.investagrams.com/InvestaApi/Stock/ViewStock?stockCode=ac&defaultExchangeType=1&cv=1622292000-0-

Using requests with some headers I can also get it.
Because it gets data as JSON so I don't need BeautifulSoup for this.

import requests

headers = { 
    'User-Agent': 'Mozilla/5.0',
    'Referer': 'https://www.investagrams.com/'
}

url = 'https://webapi.investagrams.com/InvestaApi/Stock/ViewStock?stockCode=ac&defaultExchangeType=1&cv=1622292000-0-'
r = requests.get(url, headers=headers)

#print(r.status_code)
#print(r.json())

data = r.json()
print('Last:', data['LatestStockHistory']['Last'])

for key, value in data['LatestStockHistory'].items():
    print(key, '=', value)

Result:

Last: 779

StockId = 79
Date = 2021-05-28T00:00:00+08:00
DateShortString = 05/28/2021
DateTimeString = May 28, 2021 12:00:00 AM
Last = 779
LastString = 779.00
Open = 780
Close = 800
Change = -21
ChangeString = -21.00
ChangePercentage = -2.62
ChangePercentageString = -2.62%
Low = 768
High = 789.5
Average = 778.14
Volume = 457590
Value = 356067865
Trades = 4227
MarketCap = 482.74B
NetForeign = -3091405
LastUpdateTime = 2021-05-28T15:30:00
LastUpdateTimeString = May 28, 2021 03:30:00 PM
user_18320580

The page you're trying to scrape from is using JavaScript to populate the DOM asynchronously. You can expect BeautifulSoup not to work for pages like this, because BeautifulSoup can only see what's been baked directly into the HTML at the time the server served the document to you.

If you view the page in a browser, and log your network traffic, you'll see multiple requests being made to various REST API endpoints, one of which has the endpoint /InvestaApi/Stock/ViewStock, and takes a "stock code" as a query-string parameter. The response of this endpoint is JSON, and contains the information you're trying to scrape. All you need to do is imitate that HTTP GET request:

def get_price(stock_code):
    import requests

    url = "https://webapi.investagrams.com/InvestaApi/Stock/ViewStock"

    params = {
        "stockCode": stock_code,
        "defaultExchangeType": "1",
        "cv": "1622292000-0-"
    }

    headers = {
        "accept": "application/json",
        "accept-encoding": "gzip, deflate",
        "referer": "https://www.investagrams.com/",
        "user-agent": "Mozilla/5.0"
    }

    response = requests.get(url, params=params, headers=headers)
    response.raise_for_status()

    return response.json()["LatestStockHistory"]["Last"]

def main():

    print(get_price("ac"))
    
    return 0


if __name__ == "__main__":
    import sys
    sys.exit(main())

Output:

779
>>>
user_109874320
Related