I have a "main" process and a few "worker" processes between which I want to pass some messages. The messages could be binary blobs but has a fixed size for each. I want an abstraction which will neatly buffer and separate out each message for me. I don't want to invent my own protocol on top of TCP, and I can't find any simple+lightweight solution that is portable across languages. (As of now, the "main" process is a Node.js server, and the "worker" processes are planned to be in Python.)
2 Answers
The question is purely opinion based but I will give it a shot anyway:
WebSocket is an overkill imo. First of all in order to make WebSockets work you have to implement HTTP (or at least some basic form of it) to do the handshake. If you do that then it is better to stick to "normal" HTTP unless there's a reason for full-duplex communication. There are lots of tools everywhere to handle HTTP over (unix domain) sockets.
But this might be an overkill as well. If you have workers then I suppose performance matters. The easiest and (probably) most efficient solution is a following protocol: each message starts with 1-8 (pick a number) bytes which determine the size of the following content. The content is anything you want, e.g. protobuf message. For example if you want to send foo, then you send 0x03 0x00 0x66 0x6f 0x6f. First two bytes correspond to the size of the content (being 3) and then 3 bytes correspond to foo.
It sounds like you need a message broker of some sort.
Your requirement for “buffering” would exclude, for example, ZeroMQ (which is ideal for inter-process communication, but has no built-in message persistence).
This leaves you with options such as RabbitMQ or SQS if you happen to be on AWS. You might also look at Kafka (Kinesis on AWS). These services all offer “buffering” of messages, with RabbitMQ offering the greatest range of configurations, but probably the greatest implementation hurdle.
Another option would be to use Redis as a simple messaging service.
There are a number options, all of which suit different use-cases and environments. I should add though that “simple and lightweight” doesn’t really fit with any solution other than - perhaps - ZeroMQ
Related