Bulk Upsert using DynamoDB

Viewed 94

Is it possible to do bulk upsert of items in DynamoDB similar to how MongoDB bulk write can do this?

I have this function in MongoDB looking to do something similar in DynamoDB:

async bulkSync(
    model: any | null,
    context: any[],
    filterKeys: Array<string> = ['_id'],
): Promise<any> {
    let operations = []; // Initialize the bulk operations array
    let i = 1;
    for (const record of context) {
        const filter: any = {};
        for (const key of filterKeys) {
            const path = key.split('.');
            const value = R.path(path, record);
            filter[key] = value;
        }
        operations.push({
            updateOne: {
                filter: filter,
                update: {
                $set: record,
                    $setOnInsert: {
                        created_at: record.updated_at,
                    },
                },
                upsert: true, // example update operation,
                setDefaultsOnInsert: true,
            },
        });
        if (operations.length % 500 === 0 || i === context.length) {
            await model.collection.bulkWrite(operations, {
                ordered: true,
                w: 1,
            });
            operations = [];
        }
        i++;
    }
    return;
}
1 Answers

You have two options:

  1. Use BatchWrite. In this case, DynamoDB will essentially handle concurrency for you, running multiple individual write requests in parallel. There is some overhead here in handling any failed requests.
  2. Use the PutItem request, and use node worker_threads module to implement concurrency on the client side.
Related