I'm trying to scrape a Belgian gambling website (https://circus.be) . As most of you know, on these gambling websites, you have sports betting. I'm trying to scrape the odds of the upcoming soccer games. For example the link to the Belgian league is https://sport.circus.be/en/sport/sports-bets/844/227875758 If I look in my network tab I see this page is using a websocket which connects to wss02.circus.be Having a closer look, is also giving me the message which is getting send to this host. With my little knowlegde of websockets I tried to recreate this via Python code. My code so far :
from websocket import create_connection
import json
header = json.dumps({'Date': '',
'Connection': 'upgrade',
'Origin': 'https://circus.be',
'Host': 'wss02.circus.be',
'Set-Cookie': '',
'Upgrade': 'websocket',
'Sec-WebSocket-Accept': '',
})
# Copy the web brower header and input as a dictionary
headers = json.dumps({
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'nl-NL,nl;q=0.9,en-US;q=0.8,en;q=0.7',
'Cache-Control': 'no-cache',
'Connection': 'Upgrade',
'Host': 'wss02.circus.be',
'Origin': 'https://sport.circus.be',
'Pragma': 'no-cache',
'Sec-WebSocket-Extensions': 'permessage-deflate; client_max_window_bits',
'Sec-WebSocket-Key': 'dW0WTuC2UHMdD0KugzsIKw==',
'Sec-WebSocket-Version': '13',
'Upgrade': 'websocket',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36'
})
# Launch the connection to the server.
ws = create_connection('wss://wss02.circus.be:403', headers=headers)
mts = {"Id": "bb4e52e5-d8f8-0c8d-e42b-56cd2fb57768",
"TTL": 10,
"MessageType": 1000,
"Message": "{Direction:1,Id:6310d092-d257-e740-ff32-802a934075c5,Requests:[{Id:2942dab9-c08f-c60a-9b41-a8799931dd0c,Type:201,Identifier:ContinueLeaguesDataSourceFromCache,AuthRequired:false,Content:{Entity:{CacheId:1f2eeeea-7def-4d6b-aec4-400b095ebe9f,EventIdList:[3427106501,3427106701,3427106901,9614620,9614598,9391158,9391155,9391156,9470711]},InitialRequest:{Language:en,BettingActivity:0,PageNumber:0,OnlyShowcaseMarket:true,IncludeSportList:true,EventSkip:0,EventTake:200,EventType:0,SportId:844,RequestString:LeagueIds=227875758&OnlyMarketGroup=Main}}}],Groups:[]}"}
# Perform the handshake.
ws.send(json.dumps(mts))
# Printing all the result
while True:
try:
result = ws.recv()
print(result)
except Exception as e:
print(e)
break
Unfortunately this isn't giving me a response, or the JSON with the data in it which I'm looking for. I hope someone is willing to spend a little time on this and help me out. Thanks a lot already!