Deleting collection and inserting array of docs via mongoDB Realm webhook

I have found a couple of points of interest that may help your cause. First, let me provide my solution. I used an Atlas Realm webhook with the following attributes:

• Authentication: System
• Log Function Arguments: ON
• HTTP Method: POST
• Respond With Result: ON
• Can Evaluate:
• Request Validation: No Additional Authorization

Of these items, I think the HTTP Method is the most relevant. Since you are passing data in your CURL command using the -d operator a GET method will not suffice. Your CURL command did not specify an HTTP verb so it assumes GET. The second item is request validation. I see your URL includes the special secret key item. I am not using any validation because I am focused on having the function work as expected, then I would apply security.

Web Hook Function

exports = function(payload, response) {
  
    const {database, coll_to_update} = payload.query;
    const contentTypes = payload.headers["Content-Type"];
    const body = JSON.parse(payload.body.text());
    
    // console.log("database, coll_to_update:", database, coll_to_update);
    // console.log("Content-Type:", JSON.stringify(contentTypes));
    // console.log("Request body:", body);

    const coll = context.services.get("mongodb-atlas").db(database).collection(coll_to_update);
    
    coll.deleteMany({})
     .then(result => { 
        console.log(`Deleted ${result.deletedCount} item(s).`)
       
        coll.insertMany(body)
          .then(result => console.log(`Successfully inserted ${result.insertedIds.length} items!`))
          .catch(err => console.error(`Failed to insert documents: ${err}`))
     })
     .catch(err => console.error(`Delete failed with error: ${err}`))
    
     return payload;
};

Notice the constant variable body is assigned the JSON parsed version of the .text() representation of the binary payload? This converts the $binary to a JSON object.

The second item to mention is order of operations. Your original post had the calls to the database sequentially - one after another. You first call deleteMany() then you call insertMany(). But since the code is asynchronous the call to insertMany occurred before the deleteMany and as such the inserted records never show. Instead, place the insert inside the .then() of the deleteMany().

Here is my CURL command. You must ensure the HTTP verb matches the web hook method. In my case I elected to use POST in both.

Example CURL Command

curl --verbose \
  --header "Content-Type: application/json" \
  --request POST "https://us-east-1.aws.webhooks.mongodb-realm.com/api/client/v2.0/app/barryapp-pzhuy/service/barryservice/incoming_webhook/barrywebhook?database=FIC&coll_to_update=FIC_data&secret=not_this_one" \
  --data '[ { "foo": "bar" }, { "baz": "zap" } ]'

Example mongoshell Documents

Enterprise atlas-7aocnr-shard-0 [primary]> db.FIC_data.find()
[
  { _id: ObjectId("614b649faed0b6812c95e976"), foo: 'bar' },
  { _id: ObjectId("614b649faed0b6812c95e977"), baz: 'zap' }
]

To test this out, I needed to pass a different payload and verify in the database all the records changed...

Test by issuing a second CURL command

curl --verbose \
  --header "Content-Type: application/json" \
  --request POST "https://us-east-1.aws.webhooks.mongodb-realm.com/api/client/v2.0/app/barryapp-pzhuy/service/barryservice/incoming_webhook/barrywebhook?database=FIC&coll_to_update=FIC_data&secret=not_this_one" \
  --data '[ { "foo": "abc" }, { "baz": "xyz" } ]'

Results in mongoshell

Enterprise atlas-7aocnr-shard-0 [primary]> db.FIC_data.find()
[
  { _id: ObjectId("614b6537b435654ce5212d90"), foo: 'abc' },
  { _id: ObjectId("614b6537b435654ce5212d91"), baz: 'xyz' }
]

Oh, and by the way, I confirmed with MongoDB - there is no drop() method allowed for a collection using the MongoDB Atlas Realm functions. To clear a collection you must remove all records as you are doing in your code. This is not a great solution. Another solution might be to simply abandon the collection in favor of a new collection name, maybe one with a date in the name. You could have a cron job clear out the abandoned collections with a collection.drop() command to avoid the overhead of deleting records one at a time (especially important if the collection is large on a replica set topology where the deletes are idempotent commands on the OpLog).

Generally speaking database operations are not allowed. I am unclear on all the limitations surrounding this concept but if a collection drop is not allowed I suspect index topics are also not allowed.

I managed to work it out, see the code below. This does not include the deleting of existing documents in the target collection yet.

exports = function(payload, response) {
    const {database, coll_to_update} = payload.query;        
    const coll = context.services.get("mongodb-atlas").db(database).collection(coll_to_update);

    // Payload body is a JSON string, convert into a JavaScript Object
    const data = JSON.parse(payload.body.text());

    // Perform operations as a bulk
    const bulkOp = coll.initializeOrderedBulkOp();
    data.forEach((document) => {
        bulkOp.insert(document);
    });
    response.addHeader(
        "Content-Type",
        "application/json"
    );
    bulkOp.execute().then(() => {
        // All operations completed successfully
        response.setStatusCode(200);
        response.setBody(JSON.stringify({
            timestamp: (new Date()).getTime()
        }));
        return;
    }).catch((error) => {
        // Catch any error with execution and return a 500
        response.setStatusCode(500);
        response.setBody(JSON.stringify({
            timestamp: (new Date()).getTime(),
            errorMessage: error
        }));
        return;
    });