Below are the configuration of my SQS and description of my use case
- I am pushing ~1000 messages per minute to AWS SQS and worker machine needs ~30 seconds to process a single message and uses significant amount of RAM to process that message.
- worker machine poll the SQS for every 10 Seconds and each time it may get 10 messages.
- Since for every 10 seconds a dameon in the worker, polls the SQS, so large number of messages are end up getting parallely processed and throwing memory errors.
- SQS Queue type is, Standard.
My question how to control the number of parallel messages processed by the consumer, like at any given time only 3 message should be processed parallely?
Also even if i increase the number of consumers, some consumer may end up processing a lot of messages right? because of the worker polling SQS for every 5 seconds.
I am new to this AWS SQS, kindly correct me if i am wrong.