I'm trying to consume an SSE (Server Sent Events) stream from a server using Python. To this end, I'm trying to use sseclient package.
pip install sseclient installed version 0.0.27.
Now the program:
from sseclient import SSEClient
import json
url - # URL
token = # token
headers = {}
headers['Accept'] = 'text/event-stream'
headers['Authorization'] = 'Bearer ' + token
messages = SSEClient(url, headers=headers)
for message in messages:
node = json.loads(message.data)
# process the node
When I consume a short stream (just a few messages), it works great.
But when I consume a stream consisting of a few thousand messages, it breaks:
Traceback (most recent call last):
File "/Users/romanpuchkovskiy/scripts/./search.py", line 67, in <module>
node = json.loads(message.data)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
return _default_decoder.decode(s)
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/Cellar/python@3.9/3.9.0_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 353, in raw_decode
obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 411 (char 410)
Well, actually it's further processing that breaks, but this means that sseclient corrupts the data (so it cannot be later parsed as a JSON). I checked the stream with CURL using identical request, and the response is ok. It can also be successfully consumed by a client written in Java (using WebClient from spring-webflux).
What could be the reason?