Upgrading from Python 2 > 3 - Boto's `receive_messages` blocks unrelated threads

Viewed 107

While working on upgrading some code from Python 2.7 to 3.7, I found that boto SQS client's receive_message function will block other threads while it's waiting. The following is a rough example:

sqs_client = boto_session.client('sqs')

def dequeue_message():
    logger.info('dequeue_message: BEGIN')
    time.sleep(0.5)  # Allow the other thread to start sleeping
    logger.info('dequeue_message: START POLLING')
    res = sqs_client.receive_message(
        QueueUrl=queue_url,
        MaxNumberOfMessages=1,
        VisibilityTimeout=1,
        WaitTimeSeconds=10,
    )  # This will only block if there are no messages available
    logger.info('dequeue_message: DONE {}'.format(res))

def just_wait():
    logger.info('just_wait: BEGIN')
    start_time = time.time()
    time.sleep(2)
    logger.info('just_wait: DONE (expected 2, actual {})'.format(int(time.time() - start_time)))

wait_task = threading.Thread(target=just_wait)
dequeue_task = threading.Thread(target=dequeue_message)
wait_task.start()
dequeue_task.start()
wait_task.join()
dequeue_task.join()

In python 2, the just_wait thread waits 2 seconds and returns. However, in Python 3, the just_wait thread ends up waiting 10 seconds (the wait time on receive_message) before returning, presumably somehow "blocked" by the call to receive_message.

The Python 3 behavior is a bit of a problem. What could cause it to hang like that?

I should also note that this is in a much larger tech stack that I'm not 100% familiar with. There may be other libraries or functionality running in the background that I'm not currently aware of.

Relevant library versions: boto3@1.7.84, botocore@1.10.84

0 Answers
Related