Throughput in Standard SQS vs FIFO SQS with a unique groupId for every message

user_2699001 0
Viewed 4268

I do not care much about the order of events but I would like the message to be processed exactly once. The lambda listening to SQS messages will store it in DynamoDB so throughput is pretty important as I have multiple microservices (as producers) writing messages to this SQS that will be read by a single microservice.

About processing messages exactly once, that is something that FIFO queue supports but is said to have not a good throughput.

Is the throughput of the FIFO queue the same as the Standard queue if each message has a unique groupId?

If not, my next option is probably to use "attribute_not_exists" in DynamoDB while storing the message.

Which of these should work better?

amazon-sqs amazon-dynamodb serverless serverless-framework amazon-web-services
3 Answers

FIFO SQS queues have different rate limits than a regular SQS queue regardless of the use of message group ids

SQS Standard queues support a nearly unlimited number of API calls per second, per API action (SendMessage, ReceiveMessage, or DeleteMessage).

FIFO SQS supports 300 TPS for each API method

Look at the quota docs here

Also, AWS has a new feature for higher throughput FIFO SQS queue which might interest you

With batching of maximum 10 messages per API call you can handle 3,000 messages per second with FIFO queue

Regarding making sure you don't handle the same message twice - have you had a look at FIFO de-duplication ID? I am not sure if that's exactly what you need but it sounds pretty similar to your requirement

user_15035530

Messages / sec

FIFO

  • 30,000 messages (with batching + high throughput mode)
  • 3,000 messages (without batching + high throughput mode)
  • 3,000 messages (with batching)
  • 300 messages (without batching)

Standard

  • Nearly unlimited

https://aws.amazon.com/sqs/faqs/

To process exactly once, you need to use FIFO queue with de-deplication ID.

If your throughput requirement is below the limit mentioned above, then you're fine with the FIFO queue.

If not then, using DynamoDB as your original plan is also an alternative option. But you have to manage a lot of things yourself here with this approach like deleting the message, updating if the message is being read but not yet fully processed, and so on.

user_37621130

SQS delivery guarantee is at least once. Your application must be designed to handle processing duplicate messages.

I'd strongly recommend building your application this way.

If you must process some type of data exactly once, you need a strongly consistent system. Consider using dynamodb and conditional updates

user_121608810
Related