While working on upgrading some code from Python 2.7 to 3.7, I found that boto SQS client's receive_message function will block other threads while it's waiting. The following is a rough example:
sqs_client = boto_session.client('sqs')
def dequeue_message():
logger.info('dequeue_message: BEGIN')
time.sleep(0.5) # Allow the other thread to start sleeping
logger.info('dequeue_message: START POLLING')
res = sqs_client.receive_message(
QueueUrl=queue_url,
MaxNumberOfMessages=1,
VisibilityTimeout=1,
WaitTimeSeconds=10,
) # This will only block if there are no messages available
logger.info('dequeue_message: DONE {}'.format(res))
def just_wait():
logger.info('just_wait: BEGIN')
start_time = time.time()
time.sleep(2)
logger.info('just_wait: DONE (expected 2, actual {})'.format(int(time.time() - start_time)))
wait_task = threading.Thread(target=just_wait)
dequeue_task = threading.Thread(target=dequeue_message)
wait_task.start()
dequeue_task.start()
wait_task.join()
dequeue_task.join()
In python 2, the just_wait thread waits 2 seconds and returns. However, in Python 3, the just_wait thread ends up waiting 10 seconds (the wait time on receive_message) before returning, presumably somehow "blocked" by the call to receive_message.
The Python 3 behavior is a bit of a problem. What could cause it to hang like that?
I should also note that this is in a much larger tech stack that I'm not 100% familiar with. There may be other libraries or functionality running in the background that I'm not currently aware of.
Relevant library versions: boto3@1.7.84, botocore@1.10.84