How to write 5000 records into DynamoDB Table?

user_13436208 0
Viewed 914

I have a use case where I have to write 5000 records into dynamoDB table in one shot. I am using batchSave api of DynamoDBMapper Library. it can write upto 25 records in one go. can I pass the list of 5000 records to it and it will internally convert them into batch of 25 records and write to dynamodb table or I will have to handle this thing in my code using conditional some logic and will pass only 25 records to batchSave?

dynamo-local amazon-dynamodb
2 Answers

According to the batchSave documentation, batchSave():

Saves the objects given using one or more calls to the AmazonDynamoDB.batchWriteItem

Indeed, it splits up the items you give it into appropriately-sized batches (25 items) and writes them using the DynamoDB BatchWriteItem operation.

You can see the code that does this in batchWrite() in DynamoDBMapper.java:

    /** The max number of items allowed in a BatchWrite request */
    static final int MAX_ITEMS_PER_BATCH = 25;

    // Break into chunks of 25 items and make service requests to DynamoDB
    for (final StringListMap<WriteRequest> batch :
               requestItems.subMaps(MAX_ITEMS_PER_BATCH, true)) {
        List<FailedBatch> failedBatches = writeOneBatch(batch, config.getBatchWriteRetryStrategy());
        ...
user_88912240

Here are the methods I use in order to achieve this end. I manage to do it, by first chucking the dataArray into small arrays (of length 25):

const queryChunk = (arr, size) => {
    const tempArr = []
    for (let i = 0, len = arr.length; i < len; i += size) {
        tempArr.push(arr.slice(i, i + size));
    }

    return tempArr
}

const batchWriteManyItems = async (tableName, itemObjs, chunkSize = 25) => {
    return await Promise.all(queryChunk(itemObjs, chunkSize).map(async chunk => {
        await dynamoDB.batchWriteItem({RequestItems: {[tableName]: chunk}}).promise()
    }))
}
user_19928660
Related