dynamodb ConditionExpression in Step functions

user_789782 0
Viewed 565

With the use of aws-step-functions I'm trying to putItem into dynamodb.

Here is an extract of the .yml, which works as intended.

...

AddingURLs:
  Type: Map
  ItemsPath: "$.vidPosts"
  Parameters:
    vidPostsIndex.$: "$$.Map.Item.Index" 
      vidPosts.$: "$$.Map.Item.Value"

  Iterator:
    StartAt: AddingURLIntoDynamoDB
    Comment: Looping over urls and adding them into the db

    States:
      AddingURLIntoDynamoDB:
        Type: Task
        Resource: arn:aws:states:::dynamodb:putItem
        ResultPath: null
        Parameters:
          TableName: my-table-table-name
          Item:
            videoID:
              S.$: "$.vidPosts.[0]"
            urls:
              S.$: "$.vidPosts.[1]"
          ConditionExpression: "attribute_not_exists(videoID)"    

        End: true

  Next: EndStepFunctions

EndStepFunctions:
  Comment: This marks the end of a step functions
  Type: Succeed

But the next time around, when this state is called, if it encounters an entry that already exists, because of, ConditionExpression: "attribute_not_exists(videoID)", as it should, it throws,

{
  executionArn: 'arn:aws:states:us-east-2:aaabbbccc:execution:xxxStateMachine:XXX-YYY-ZZZ',
  stateMachineArn: 'arn:aws:states:us-east-2:aaabbbccc:stateMachine:xxxStateMachine',
  name: '9814ce93-bc42-4f31-a742-089ea7db9ad6',
  status: 'FAILED',
  startDate: 2021-03-18T15:30:45.912Z,
  stopDate: 2021-03-18T15:30:52.386Z,
  input: '{}',
  inputDetails: { included: true },
  error: 'DynamoDB.ConditionalCheckFailedException',
  cause: 'The conditional request failed (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException; Request ID: bfd76f51-567b-439c-9a39-5a1643e75a69; Proxy: null)'
}

My question is within aws-step-functions, how do I make it ignore existing entries but continue with the iterator?

Appreciate any help and thank you in advance.

aws-step-functions amazon-dynamodb
2 Answers

We just need to catch DynamoDB.ConditionalCheckFailedException exception and move forward, without throwing any error.

What we need is

  "Catch": [
    {
      "ErrorEquals": [
        "DynamoDB.ConditionalCheckFailedException"
      ],
      "Next": "FinalStep"
    }
  ]

Here is full step function.

{
  "StartAt": "put-item",
  "States": {
    "put-item": {
      "Next": "FinalStep",
      "Catch": [
        {
          "ErrorEquals": [
            "DynamoDB.ConditionalCheckFailedException"
          ],
          "Next": "FinalStep"
        }
      ],
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:putItem",
      "Parameters": {
        "Item": {
          "pk": {
            "S": "1"
          },
          "test": {
            "S": "value"
          }
        },
        "TableName": "test",
        "ConditionExpression": "attribute_not_exists(#videoID)",
        "ExpressionAttributeNames": {
          "#videoID": "videoID"
        }
      }
    },
    "FinalStep": {
      "Type": "Pass",
      "End": true
    }
  }
}

Put-item failed with error and still moved to next task.

{
  "Error": "DynamoDB.ConditionalCheckFailedException",
  "Cause": "The conditional request failed (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ConditionalCheckFailedException; Request ID: xxxx-xxx-xxx-xxx-xxxxx; Proxy: null)"
}

enter image description here

user_101868080

I think you have a few options here.

  1. Don't put a condition on the putItem. If the data is the same then you can just allow it to override the data with the new data, which won't really change anything. Keep in mind that putItem is not the same as an SQL INSERT. It's more like an UPSERT.
  2. Add a Catch block to your step and have it proceed to the same place as the success case. You may need to have it first go to a Pass state if you are expecting something from the output of the putItem to be in the input of the next step.
  3. Add a step before the put that does a get and then do the put on a conditional step. This may not be the best option if there are cases where you could get multiple invocations for the same record within close proximity because you could have both hit the get and not see a record and then both would do a put (in which case one would fail). If you won't ever have that case then this would work.
user_28600320
Related