I've been arguing with my programmer about the best way of going about this. We have data that comes in at a rate of about 10000 objects per second. This needs to be processed asynchronously, but loose ordering is sufficient, so each object is inserted round-robin-ly into one of several message queues (there are also several producers and consumers). Each object is ~300 bytes. And it needs to be durable, so the MQs are configured to persist to disk.
The problem is that often these objects are duplicated (as in they are unavoidably duplicated in the data that comes in to the producer). They do have 10-byte unique ids. It's not catastrophic if objects are duplicated in the queue, but it is if they're duplicated in the processing after being taken from the queue. What's the best way to go about ensuring as close as possible to linear scalability whilst ensuring there's no duplication in the processing of the objects? And perhaps linked to that, should the whole object be stored in the message queue, or only the id with the body stored in something like cassandra?
Thank you!
Edit: Confirmed where the duplication occurs. Also, so far I've had 2 recommendations for Redis. I'd previously been considering RabbitMQ. What are the pros and cons of each with regards to my requirements?