Python, Scraping BS4

Mats Seamont 0
Viewed 42

There are a lot of post about this subject but I still don't manage to achieve what I want so here is my problem:

I am trying to extract stock price from this site: https://bors.e24.no/#!/instrument/NHY.OSE

and I would like extract the price: 57,12 from the "inspection" text:

<div class="number LAST" data-reactid=".g.1.2.0">
57,12</div>

Here is the code I tried which generate "AttributeError" and 'NoneType' object has no attribute 'text'.

I also tried to remove .text, in the PRICE line, and the result is 'Price is: None'

from bs4 import BeautifulSoup
import requests
url = 'https://bors.e24.no/#!/instrument/NHY.OSE'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'lxml')
PRICE= soup.find('div', class_= "number LAST").text
print('Price is:',(PRICE))
attributeerror beautifulsoup web-scraping python
2 Answers

Try this:

import requests

headers = {
    'user_agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36',
}

api_url = "https://bors.e24.no/server/components?columns=ITEM, LAST, BID, ASK, CHANGE, CHANGE_PCT, TURNOVER, LONG_NAME&itemSector=NHY.OSE&type=table"
data = requests.get(api_url, headers=headers).json()
print(data["rows"][0]["values"]["LAST"])

Output:

56.92
baduker0

This happens because your

requests.get(url)

Will not get all information in the page, including the price you are looking for, because the said webpage will load some parts of it and only then fetch more data. Because of that, trying to select the div with className="number LAST"

PRICE= soup.find('div', class_= "number LAST").text

Will throw an error because this doesn't exist, yet.

There are some ways to fix this problem:

  1. You can try to use libraries like Selenium, which is often recommended for scraping more dynamic pages that rely on some Javascript and API calls to load content.

  2. You can open your developer tools and inspect the Network tab where you might find the request that fetches the price you are trying to scrap.

I believe that in your case, after taking a look at the Network tab myself, the right URL to request could be 'https://bors.e24.no/server/components?columns=TIME,+PRICE,+VOLUME,+BUYER,+SELLER,+ID&filter=ITEM%3D%3DsNHY&limit=5&source=feed.ose.trades.EQUITIES%2BPCC&type=history', which seems to return a dictionary with the price you are looking for.

import requests
url = 'https://bors.e24.no/server/components?columns=TIME,+PRICE,+VOLUME,+BUYER,+SELLER,+ID&filter=ITEM%3D%3DsNHY&limit=5&source=feed.ose.trades.EQUITIES%2BPCC&type=history'
page = requests.get(url)
print(page.json()["rows"][0]["values"]["PRICE"])

If you are looking to scrap various links, you will need to find a way to dynamically change the previous link to one that matches others that you are trying to crawl. Which I guess would mean to change "NHY" and "ose" to something that would match other stock that you are looking for.

Bernardo Alves0
Related