DynamoDB BatchWriteItem: Provided list of item keys contains duplicates

user_1520421 0
Viewed 22544

I am trying to use DynamoDB operation BatchWriteItem, wherein I want to insert multiple records into one table.

This table has one partition key and one sort key.

I am using AWS lambda and Go language.

I get the elements to be inserted into a slice.

I am following this procedure.

  1. Create PutRequest structure and add AttributeValues for the first record from the list.

  2. I am creating WriteRequest from this PutRequest

  3. I am adding this WriteRequest to an array of WriteRequests

  4. I am creating BatchWriteItemInput which consists of RequestItems, which is basically a map of Tablename and the array of WriteRequests.

After that I am calling BatchWriteItem, which results into an error:

Provided list of item keys contains duplicates.

Any pointers, why this could be happening?

amazon-dynamodb aws-lambda amazon-web-services
3 Answers

You've provided two or more items with identical partition/sort keys.

Per the BatchWriteItem docs, you cannot perform multiple operations on the same item in the same BatchWriteItem request.

user_2714150

Consideration: This answers works for Python

As @Benoit has remarked, the boto3 documentation states:

If you want to bypass no duplication limitation of single batch write request as botocore.exceptions.ClientError: An error occurred (ValidationException) when calling the BatchWriteItem operation: Provided list of item keys contains duplicates.

you could specify overwrite_by_pkeys=['partition_key', 'sort_key'] on the batch writer to "de-duplicate request items in buffer if match new request item on specified primary keys" according to the documentation and the source code. That is, if the combination primary-sort already exists in the buffer it will drop that request and replace it with the new one.

Example

Suppose there is pandas dataframe that you want to write to a DynamoDB table, the following function could be helpful,

import json 
import datetime as dt 
import boto3 
import pandas as pd 
from typing import Optional

def write_dynamoDB(df:'pandas.core.frame.DataFrame', tbl:str, partition_key:Optional[str]=None, sort_key:Optional[str]=None):
    '''
       Function to write a pandas DataFrame to a DynamoDB Table through 
       batchWrite operation. In case there are any float values it handles 
       them by converting the data to a json format.  
       Arguments: 
       * df: pandas DataFrame to write to DynamoDB table. 
       * tbl: DynamoDB table name. 
       * partition_key (Optional): DynamoDB table partition key. 
       * sort_key (Optional): DynamoDB table sort key. 
    '''

    # Initialize AWS Resource 
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table(tbl)

    # Check if overwrite keys were provided
    overwrite_keys = [partition_key, sort_key] if partition_key else None

    # Check if they are floats (convert to decimals instead) 
    if any([True for v in df.dtypes.values if v=='float64']):

        from decimal import Decimal

        # Save decimals with JSON
        df_json = json.loads(
                       json.dumps(df.to_dict(orient='records'),
                                  default=date_converter,
                                  allow_nan=True), 
                       parse_float=Decimal
                       )

        # Batch write 
        with table.batch_writer(overwrite_by_pkeys=overwrite_keys) as batch: 
            for element in df_json:
                batch.put_item(
                Item=element
            )

    else: # If there are no floats on data 

    # Batch writing 
        with table.batch_writer(overwrite_by_pkeys=overwrite_keys) as batch: 

            columns = df.columns

            for row in df.itertuples():
                batch.put_item(
                    Item={
                        col:row[idx+1] for idx,col in enumerate(columns)
                    }
                )

def date_converter(obj):
    if isinstance(obj, dt.datetime):
        return obj.__str__()
    elif isinstance(obj, dt.date):
        return obj.isoformat()

by calling write_dynamoDB(dataframe, 'my_table', 'the_partition_key', 'the_sort_key').

user_124833460

Use batch_writer instead of batch_write_item:

import boto3

dynamodb = boto3.resource("dynamodb", region_name='eu-west-1')
my_table = dynamodb.Table('mirrorfm_yt_tracks')

with my_table.batch_writer(overwrite_by_pkeys=["user_id", "game_id"]) as batch:
    for item in items:
        batch.put_item(
            Item={
                'user_id': item['user_id'],
                'game_id': item['game_id'],
                'score': item['score']
            }
        )

If you don't have a sort key, overwrite_by_pkeys can be None

This is essentially the same answer as @MiguelTrejo (thanks! +1) but simplified

user_15158190
Related