I've been working already with pyzmq for one year. Actually facing an issue with a websocket providing very high volume of trading data. My original design was as follows (running Python 3.8 and pyzmq under macos Catalina):
- subscribe to the symbols I want to follow
- for every subscribed symbol, add an entry in a dictionary where the keys are these symbol-names
- add a zmq.PAIR socket to that dictionary entry, which will operate as a Client
- for every incoming message, extract the symbol for that message
- write the message to the client socket pointed by the proper dictionary key
- another process (which has already been launched with another zmq.PAIR opened as a Server) receives the data and processes it.
After some weeks operating in test, and after to properly debug issues with my app, during live operation and after some time of execution, the socket is being silently closed by my data provider.
While I did not yet addressed much research on my own environment, I started looking for a message-queue system as a possible solution after I found my same problem was reported here: https://github.com/polygon-io/issues/issues/125
Should I have to read the entire AMQP definition to understand what I need to do?
I am primarily focused on avoiding a big message broker but instead starting simple, adding parts as I see they are needed.
Would like to know if there are already any efforts in this direction, to avoid reinventing the wheel if possible.
I've been using inproc and PAIR zmq sockets with high success, and also coded my own async version of a REQ/REP client-server layer with high success too.
Thanks very much in advance for your help