Constantly polling SQS Queue using infinite loop

user_803533 0
Viewed 2027

I have an SQS queue that I need to constantly monitor for incoming messages. Once a message arrives, I do some processing and continue to wait for the next message. I achieve this by setting up an infinite loop with a 2 second pause at the end of the loop. This works however I can't help but feel this isn't a very efficient way of solving the need to constantly pole the queue.

Code example:

    while (1):
        response = sqs.receive_message(
            QueueUrl=queue_url,
            AttributeNames=[
                'SentTimestamp'
            ],
            MaxNumberOfMessages=1,
            MessageAttributeNames=[
                'All'
            ],
            VisibilityTimeout=1,
            WaitTimeSeconds=1
        )

        try:
            message = response['Messages'][0]
            receipt_handle = message['ReceiptHandle']

            # Delete received message from queue
            sqs.delete_message(
                QueueUrl=queue_url,
                ReceiptHandle=receipt_handle
            )
            msg = message['Body']
            msg_json = eval(msg)
            value1 = msg_json['value1']
            value2 = msg_json['value2']
            process(value1, value2)
        except:
            pass
            #print('Queue empty')
        time.sleep(2)

In order to exit the script cleanly (which should run constantly), I catch the KeyboardInterrupt which gets triggered on Ctrl+C and do some clean-up routines to exit gracefully.

if __name__ == '__main__':
    try:
        main()
    except KeyboardInterrupt:
        logout()

Is there a better way to achieve the constant poling of the SQS queue and is the 2 second delay necessary? I'm trying not to hammer the SQS service, but perhaps it doesn't matter?

amazon-sqs python
2 Answers

This is ultimately the way that SQS works - it requires something to poll it to get the messages. But some suggestions:

Don't get just a single message each time. Do something more like:

messages = sqs.receive_messages(
        MessageAttributeNames=['All'],
        MaxNumberOfMessages=10,
        WaitTimeSeconds=10
    )
for msg in messages:
    logger.info("Received message: %s: %s", msg.message_id, msg.body)

This changes things a bit for you. The first thing is that you're willing to get up to 10 messages (this is the maximum number for SQS in one call). The second is that you will wait up to 10 seconds to get the messages. From the SQS docs:

The duration (in seconds) for which the call waits for a message to arrive in the queue before returning. If a message is available, the call returns sooner than WaitTimeSeconds. If no messages are available and the wait time expires, the call returns successfully with an empty list of messages.

So you don't need your own sleep call - if there are no messages the call will wait until it expires. Conversely, if you have a ton of messages then you'll get them all as fast as possible as you won't have your own sleep call in the code.

user_29339770

Adding on @stdunbar Answer:

You will find that MaxNumberOfMessages as stated by the Docs might return fewer messages than the provided integer number, which was the Case for me.

MaxNumberOfMessages (integer) -- The maximum number of messages to return. Amazon SQS never returns more messages than this value (however, fewer messages might be returned). Valid values: 1 to 10. Default: 1.

As a result, i made this solution to read from SQS Dead-Letter-Queue:

def read_dead_letter_queue():

""" This function is responsible for Reading Query Execution IDs related to the insertion that happens on Athena Query Engine
         and we weren't able to deal with it in the Source Queue.

Args: 
    None

Returns:
    Dictionary: That consists of execution_ids_list, mssg_receipt_handle_list and queue_url related to messages in a Dead-Letter-Queue that's related to the insertion operation into Athena Query Engine.
"""

  try:
    sqs_client = boto3.client('sqs')
    
    queue_url = os.environ['DEAD_LETTER_QUEUE_URL'] 
    
    execution_ids_list = list()
    
    mssg_receipt_handle_list = list()
    
    final_dict = {}
    
    # You can change the range stop number to whatever number that suits your scenario, you just need to add a number that's more than the number of messages that maybe in the Queue as 1 thousand or 1 million, as the loop will break out when there aren't any messages left in the Queue before reaching the end of the range.
    for mssg_counter in range(1, 20, 1):
        
        sqs_response = sqs_client.receive_message(
            QueueUrl = queue_url,
            MaxNumberOfMessages = 10,
            WaitTimeSeconds = 10
        )
        
        print(f"This is the dead-letter-queue response --> {sqs_response}")
        
        try:
            for mssg in sqs_response['Messages']:
                    print(f"This is the message body --> {mssg['Body']}")
                    print(f"This is the message ID --> {mssg['MessageId']}")
                    execution_ids_list.append(mssg['Body'])
                    mssg_receipt_handle_list.append(mssg['ReceiptHandle'])
        except:
            print(f"Breaking out of the loop, as there isn't any message left in the Queue.")
            break
    
    print(f"This is the execution_ids_list contents --> {execution_ids_list}")
    
    print(f"This is the mssg_receipt_handle_list contents --> {mssg_receipt_handle_list}")
    
    # We return the ReceiptHandle to be able to delete the message after we read it in another function that's responsible for deletion.
    # We return a dictionary consists of --> {execution_ids_list: ['query_exec_id'], mssg_receipt_handle_list: ['ReceiptHandle']}
    
    final_dict['execution_ids_list'] = execution_ids_list
    final_dict['mssg_receipt_handle_list'] = mssg_receipt_handle_list
    final_dict['queue_url'] = queue_url
    
    return final_dict
    
    #TODO: We need to delete the message after we finish reading in another function that will delete messages for both the DLQ and the Source Queue.
        
    
  except Exception as ex:
    print(f"read_dead_letter_queue Function Exception: {ex}")
user_45431680
Related