How can I reliably tell when an AWS RDS modification operation is complete?

user_3248882 0
Viewed 648

I'm using boto3 in a small script to change the master credentials of an RDS cluster.

The issue is that the RDS cluster status from describe_db_clusters isn't reliably up to date. After I issue a modify_db_cluster (specifically, I'm updating the master password), there is a high chance that describe_db_clusters shows that the cluster is still "available", before it enters the "modifying" state.

The sequence of events:

  1. cluster is available
  2. modify cluster password
  3. cluster is still available
  4. cluster is modifying
  5. cluster is available (operation complete)

The issue is that I can't reliably tell between states 3 and 5.

Notes:

  • I'm using ApplyImmediately=True for the master password update
  • I see the same problematic behaviour, whether I inspect the cluster status, or the cluster's PendingModifiedValues

The docs do mention this for describe_db_instances:

MasterUserPassword (string) -- The new password for the master user. The password can include any printable ASCII character except "/", """, or "@". Changing this parameter doesn't result in an outage and the change is asynchronously applied as soon as possible. Between the time of the request and the completion of the request, the MasterUserPassword element exists in the PendingModifiedValues element of the operation response.

However, that footnote doesn't exist for describe_db_clusters.

How can I reliably know when my modification operation is complete?

boto3 amazon-rds amazon-web-services
2 Answers

Apparently, only if there is a pending change, PendingModifiedValues will be present in describe_db_clusters response:

#!/usr/bin/python3
import boto3
import time
import datetime
client = boto3.client('rds',region_name='ap-southeast-2')
while True:
    response = client.describe_db_clusters(
            Filters=[
            {
                'Name': 'db-cluster-id',
                'Values': [
                    'database-1',
                ]
            }
        ]
    )
    for cluster in response['DBClusters']:
        print(cluster['DBClusterIdentifier'],cluster['Status'])
        if "PendingModifiedValues" in cluster:
            print(cluster['PendingModifiedValues'])
        else:
            print('No PendingModifiedValues in response')
    now = datetime.datetime.now()
    print(now.strftime("%Y-%m-%d %H:%M:%S"))
    time.sleep(1)

Output:

$ ./check_rds_cluster.py 

database-1 available
No PendingModifiedValues in response
2021-03-28 13:19:45
database-1 available
No PendingModifiedValues in response
2021-03-28 13:19:46
database-1 available
No PendingModifiedValues in response
2021-03-28 13:19:48
database-1 available
{'MasterUserPassword': '****'}
2021-03-28 13:19:49
...
database-1 available
{'MasterUserPassword': '****'}
2021-03-28 13:19:53
database-1 available
{'MasterUserPassword': '****'}
2021-03-28 13:19:55
database-1 resetting-master-credentials
{'MasterUserPassword': '****'}
2021-03-28 13:19:56
database-1 resetting-master-credentials
{'MasterUserPassword': '****'}
...
2021-03-28 13:21:17
database-1 available
No PendingModifiedValues in response
2021-03-28 13:21:18
database-1 available
No PendingModifiedValues in response
2021-03-28 13:21:19

It is documented in https://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_DBCluster.html:

PendingModifiedValues

A value that specifies that changes to the DB cluster are pending. This element is only included when changes are pending. Specific changes are identified by subelements.

user_11451960

In such cases, you could create a waiter function in your code. The boto3 RDS provides a couple of waiters already, but none applies to a cluster. The closest is DBInstanceAvailable.

Basically, a waiter is a function that pulls AWS API iteratively with 15 seconds delays to check if a given resource is in a desired state, e.g. RDS instance has been created and is available for use.

So you could create your own waiter which would pull describe_db_clusters ever 10 or 15 seconds until PendingModifiedValues is gone from the response or has some other state of interest. Maybe checking DBClusters's Status is enough.

user_2488230
Related