boto3 put_item succeeds but record does not appear

Viewed 679

I haven't been able to find an answer for this anywhere, hoping SO might be able to finally help.

I've got a lambda function that processes a record then writes it to a dynamodb table. By all intents and purposes, it appears that the put_item call is succeeding, however, when I check the dynamo table the record is not appearing in it.

import json
import boto3
import uuid
import urllib3
from botocore.exceptions import ClientError

def lambda_handler(event, context):
  
  apiResponse = {}
  for record in event['Records']:
    decoded = json.loads(record['body'])
    listId = int(decoded['queryParams']['rec_id'])
    apiCall = "INTERNAL API"
    http = urllib3.PoolManager()
    request = http.request('GET', apiCall)
    apiResponse = json.loads(request.data.decode('utf-8'))

    try:
      client = boto3.resource('dynamodb')
      table = client.Table('HistoryAuditTable')
      saveStatus = table.put_item(Item={
        'UUID': uuid.uuid4().hex,
        'RecId': listId,
        'MessageType': decoded['queryParams']['type'],
        'MessageTimestampUTC': record['attributes']['SentTimestamp'],
        'Message': apiResponse
      })
      print("STATUS")
      print(saveStatus) # This prints out a 200 status code in CloudWatch
    except ClientError as e:
      # This error never happens.
      print("ERROR")
      print(e.response["Error"]['Message'])
      
  # Response Status
  response = {}
  response["body"] = json.dumps(decoded)

  return response

My apiResponse is a simple json payload containing some audit data that we keep track of whenever a record changes.

My dynamo table has the following fields:

UUID: self-explanatory, 
RecId: an internal record identifier (we keep this separate from the PK because the same record might be updated again and again),
MessageType: String representing if the record was a "SEED_VALUE", "CREATE", "UPDATE", "ARCHIVE", or "SOFT_DELETE"
MessageTimestampUTC: self-explanatory
Message: JSON blob containing the record details

I had to seed this table with an initial load of the current state of our data (around 400k records) the seeding process used this same aws lambda function. My first indication that something was up was that only around 100k of the 400k records actually made it into the table even though all 400k rows returns a 200 response code, after I noticed this I decided to try and trigger the processes which would push individual records into the table, I can see that my function is firing correctly, I can see that my api calls are returning correctly, and I can see that the saveStatus appears to be successful, however I am not seeing my records in the dynamo table. I guess I have a couple questions here:

  1. Is there some write limit for dynamodb tables? Did I possibly push too much data into it initially and I simply can't push anything new until my limit resets? (I can't find a straight answer to this anywhere online)
  2. Am I doing something completely wrong? My process was copied almost exactly from the AWS documentation but AWS's documentation is notoriously bad.
  3. Is there some sort of error/write logging which can be enabled for dynamodb through cloudwatch? (I can't find any information on this either anywhere)

A couple other things I've checked:

  • The UUIDs aren't colliding (unlikely I know but I've become desperate)
  • The individual records are WELL under the 400kb DynamoDB limit (the largest record I've seen is 5kb)

I'm honestly at a loss, I don't get how/why the process worked for 100k records then all of a sudden decided to not work anymore.

1 Answers

Some recommendations, pointers and observations in general, this might get more specific if you answer the questions in the comment.

Take a look at the read and write capacity modes as well as partitioning in DynamoDB. Essentially there is an upper limit of 1000 Writes per Partition of 1KB each. Depending on your key-setup you may hit that. You can find out if that's the case in the metrics of your table. Also: you'll get exceptions for failed writes, which brings me to my next point.

In your code you essentially treat the DynamoDB write as nice-to-have. If it succeeds, that's great, if not - you just log the error and continue. That's not an ideal setup for debugging and in that case you may consider offloading the writes to DynamoDB into a separate Lambda, by storing them in SQS for example and processing them in batches. If the writes are important to your client, don't "silence" the error, raise an appropriate exception and signal to the client that something went wrong. Looking for the odd Error in the logs of a million successes isn't fun ;-)

The good news is, you're not doing it completely wrong, but you could optimize some stuff. Instantiating boto3 resources and clients is a relatively costly operation, if you care about performance, you could cache those. Consider using batching to reduce the amount of network requests.

You can monitor the amount of potential throttles as well as read/write errors in the Metrics-Tab of your table. The metrics are called:

  • Throttled [write|read] requests
  • Throttled [write|read] events
  • System errors [write|read]
Related