Scraping data over websockets

user_13756247 0
Viewed 1270

I am trying to get the daily price data from this specific webpage:

https://www.londonstockexchange.com/stock/CS1/amundi/company-page

Those data are represented in the chart.

I run out of idea to try to reach those data. I assume that those data are transfered though one of the websocket connection that is made and retrievable in the browser console.

enter image description here

I tried to simulate the websocket connection and send the same binary than the front app.

from websocket import create_connection

s = create_connection("wss://82-99-29-151.infrontservices.com/wsrt/2/4")

hex_1 = "3e000000010..."
hex_2 = "13000000010..."
hex_3 = "1e000000010..."

ws.send(binascii.unhexlify(hex_1))
ws.send(binascii.unhexlify(hex_2))
ws.send(binascii.unhexlify(hex_3))

result =  ws.recv()

Then I tried to decode this response with all the possible encoding as follow:

import binascii
from encodings.aliases import aliases

for v in [v for k, v in aliases.items()]:
    try:
        print(result.decode(v))
    except:
        print(f"ERROR {v}")

And naturally, I have no interpretable output that I can exploit. I could think that a cipher is used here. But I have no more idea how to investigate further.

Do you have any idea about that? :)

Thanks in advance !

AL Ko

EDIT 1

enter image description here

We can see one the datapoint with the value 16990 for a given date. This is what I am looking for is the whole time series of the chart.

scrapy web-scraping encoding websocket python
1 Answers

After you read my comment and get informed about scraping, and decide to proceed carefully,

Python can retrieve this JSON with just a few lines of code

import requests
url = "https://api.londonstockexchange.com/api/gw/lse/instruments/alldata/CS1"
response = requests.get(url=url).json()
# print some data from the json
print(response_json)
print(response_json.get("description"))
print(response_json.get("bid"))

I found this data using the "network" tab, a few more show up when you hit "reload", but they seem to be empty.

user_118036870
Related