AWS Aurora Serverless - Communication Link Failure

Here is the reply from AWS Premium Business Support.

Summary: It is an expected behavior

Detailed Answer:

I can see that you receive this error when your Aurora Serverless instance is inactive and you stop receiving it once your instance is active and accepting connection. Please note that this is an expected behavior. In general, Aurora Serverless works differently than Provisioned Aurora , In Aurora Serverless, while the cluster is "dormant" it has no compute resources assigned to it and when a db. connection is received, Compute resources are assigned. Because of this behavior, you will have to "wake up" the clusters and it may take a few minutes for the first connection to succeed as you have seen.

In order to avoid that you may consider increasing the timeout on the client side. Also, if you have enabled Pause, you may consider disabling it [2]. After disabling Pause, you can also adjust the minimum Aurora capacity unit to higher value to make sure that your Cluster always having enough computing resource to serve the new connections [3]. Please note that adjusting the minimum ACU might increase the cost of service [4].

Also note that Aurora Serverless is only recommend for certain workloads [5]. If your workload is highly predictable and your application needs to access the DB on a regular basis, I would recommend you use Provisioned Aurora cluster/instance to insure high availability of your business.

[2] How Aurora Serverless Works - Automatic Pause and Resume for Aurora Serverless - https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.how-it-works.html#aurora-serverless.how-it-works.pause-resume

[3] Setting the Capacity of an Aurora Serverless DB Cluster - https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.setting-capacity.html

[4] Aurora Serverless Price https://aws.amazon.com/rds/aurora/serverless/

[5] Using Amazon Aurora Serverless - Use Cases for Aurora Serverless - https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html#aurora-serverless.use-cases

If it is useful to someone this is how I manage retries while Aurora Serverless wake up.

Client returns a BadRequestException so boto3 will not retry even if you change the config for the client, see https://boto3.amazonaws.com/v1/documentation/api/latest/guide/retries.html.

My first option was to try with Waiters but RDSData does not have any waiter, then I tried to create a custom Waiter with an Error matcher but only tries to match error code, ignoring message, and because a BadRequestException can be raised by an error in a sql statement I needed to validate message too, so I using a kind of waiter function:

def _wait_for_serverless():
    delay = 5
    max_attempts = 10

    attempt = 0
    while attempt < max_attempts:
        attempt += 1

        try:
            rds_data.execute_statement(
                database=DB_NAME,
                resourceArn=CLUSTER_ARN,
                secretArn=SECRET_ARN,
                sql_statement='SELECT * FROM dummy'
            )
            return
        except ClientError as ce:
            error_code = ce.response.get("Error").get('Code')
            error_msg = ce.response.get("Error").get('Message')

            # Aurora serverless is waking up
            if error_code == 'BadRequestException' and 'Communications link failure' in error_msg:
                logger.info('Sleeping ' + str(delay) + ' secs, waiting RDS connection')
                time.sleep(delay)
            else:
                raise ce

    raise Exception('Waited for RDS Data but still getting error')

and I use it in this way:

def begin_rds_transaction():
    _wait_for_serverless()

    return rds_data.begin_transaction(
        database=DB_NAME,
        resourceArn=CLUSTER_ARN,
        secretArn=SECRET_ARN
    )

I also got this issue, and taking inspiration from the solution used by Arless and the conversation with Jimbo, came up with the following workaround.

I defined a decorator which retries the serverless RDS request until the configurable retry duration expires.

import logging
import functools
from sqlalchemy import exc
import time

logger = logging.getLogger()


def retry_if_db_inactive(max_attempts, initial_interval, backoff_rate):
    """
    Retry the function if the serverless DB is still in the process of 'waking up'.
    The configration retries follows the same concepts as AWS Step Function retries.
    :param max_attempts: The maximum number of retry attempts
    :param initial_interval: The initial duration to wait (in seconds) when the first 'Communications link failure' error is encountered
    :param backoff_rate: The factor to use to multiply the previous interval duration, for the next interval
    :return:
    """

    def decorate_retry_if_db_inactive(func):

        @functools.wraps(func)
        def wrapper_retry_if_inactive(*args, **kwargs):
            interval_secs = initial_interval
            attempt = 0
            while attempt < max_attempts:
                attempt += 1
                try:
                    return func(*args, **kwargs)

                except exc.StatementError as err:
                    if hasattr(err.orig, 'response'):
                        error_code = err.orig.response["Error"]['Code']
                        error_msg = err.orig.response["Error"]['Message']

                        # Aurora serverless is waking up
                        if error_code == 'BadRequestException' and 'Communications link failure' in error_msg:
                            logger.info('Sleeping for ' + str(interval_secs) + ' secs, awaiting RDS connection')
                            time.sleep(interval_secs)
                            interval_secs = interval_secs * backoff_rate
                        else:
                            raise err
                    else:
                        raise err

            raise Exception('Waited for RDS Data but still getting error')

        return wrapper_retry_if_inactive

    return decorate_retry_if_db_inactive

which can then be used something like this:

@retry_if_db_inactive(max_attempts=4, initial_interval=10, backoff_rate=2)
def insert_alert_to_db(sqs_alert):
    with db_session_scope() as session:
        # your db code
        session.add(sqs_alert)

    return None

Please note I'm using sqlalchemy, so the code would need tweaking to suit specific purposes, but hopefully will be useful as a starter.