MongoDB aggregation : Keep value from previous document if null or not exists

There is no straight way to do this operation,

• $map to iterate loop of log array, check condition if connection_quality type is missing then go to select previous connection_quality otherwise return the current object
• $filter to iterate loop of log and by conditions are: dt should less than and connection_quality should not missing
• now we have to select the latest connection_quality from above filtered result so using $last we will select last object
• $let to declare a variable and do above filter operation and return just connection_quality value
• $unwind to deconstruct the log array
• $project to project the result as per your requirement

db.collection.aggregate([
  {
    $match: {
      conf: "conference123",
      uid: "2dd8b4e3-9dcd-4da6-bc36-aa0988dc9642"
    }
  },
  {
    $addFields: {
      log: {
        $map: {
          input: "$log",
          as: "l",
          in: {
            $cond: [
              { $eq: [{ $type: "$$l.connection_quality" }, "missing"] },
              {
                dt: "$$l.dt",
                connection_quality: {
                  $let: {
                    vars: {
                      log: {
                        $last: {
                          $filter: {
                            input: "$log",
                            cond: {
                              $and: [
                                { $lt: ["$$this.dt", "$$l.dt"] },
                                {
                                  $ne: [{ $type: "$$this.connection_quality" }, "missing"]
                                }
                              ]
                            }
                          }
                        }
                      }
                    },
                    in: "$$log.connection_quality"
                  }
                }
              },
              "$$l"
            ]
          }
        }
      }
    }
  },
  { $unwind: "$log" },
  {
    $project: {
      _id: 0,
      "Date": "$log.dt",
      "User ID": "$uid",
      "Connection Quality": "$log.connection_quality"
    }
  }
])

Playground

You can do it with reduce The bellow query adds the connection_quality if null or missing with the value of the previous member that had connection_quality

It starts with a default 60, for example if the first member didn't had also

db.collection.aggregate([
  {
    "$addFields": {
      "log": {
        "$arrayElemAt": [
          {
            "$reduce": {
              "input": "$log",
              "initialValue": [
                60,
                []
              ],
              "in": {
                "$let": {
                  "vars": {
                    "prv_value_logs": "$$value",
                    "log": "$$this"
                  },
                  "in": {
                    "$let": {
                      "vars": {
                        "prv_value": {
                          "$arrayElemAt": [
                            "$$prv_value_logs",
                            0
                          ]
                        },
                        "logs": {
                          "$arrayElemAt": [
                            "$$prv_value_logs",
                            1
                          ]
                        }
                      },
                      "in": {
                        "$cond": [
                          {
                            "$and": [
                              {
                                "$ne": [
                                  "$$log.connection_quality",
                                  null
                                ]
                              },
                              {
                                "$ne": [
                                  {
                                    "$type": "$$log.connection_quality"
                                  },
                                  "missing"
                                ]
                              }
                            ]
                          },
                          [
                            "$$log.connection_quality",
                            {
                              "$concatArrays": [
                                "$$logs",
                                [
                                  "$$log"
                                ]
                              ]
                            }
                          ],
                          [
                            "$$prv_value",
                            {
                              "$concatArrays": [
                                "$$logs",
                                [
                                  {
                                    "$mergeObjects": [
                                      "$$log",
                                      {
                                        "connection_quality": "$$prv_value"
                                      }
                                    ]
                                  }
                                ]
                              ]
                            }
                          ]
                        ]
                      }
                    }
                  }
                }
              }
            }
          },
          1
        ]
      }
    }
  }
])

Test code here

It doesn't change the document, just adds the missing connection_quality, if you want to change it after, you can add more stages.

Solution is fast for arrays <500 members.

Edit1

The slow part is not the $reduce its the $concat because MongodDB doesn't have a way to add 1 element to the end fast. Its not how many arrays you have, but how big they are.

I was curious why you said you cant use reduce and map/filter worked for you(because looks like O(n^2)), so i did a benchmark.

1000 elements (the log)
"Elapsed time: 44.408292 msecs"   //reduce 
"Elapsed time: 167.653179 msecs"  //map and filter 3x

5000 elements
"Elapsed time: 263.549371 msecs"   
"Elapsed time: 3298.880892 msecs"  //10x+

10000 elements
"Elapsed time: 996.340296 msecs"
"Elapsed time: 14765.732331 msecs" //10x+

This is only for 1 document collection, so both solutions are very slow, not usable for big collections with big arrays > 500 elements.