How do you `$push` to a nested MongoDB array conditionally so that there are never 2 consecutive values the same?

Viewed 91

Data structure

I collect readings from probes located at different locations. Readings are recorded as {'value': <float>, 'when': <timestamp>} in an array of readings and are ordered in ascending time order. The readings array is in a subdocument accessed by probe_id.

{'location_id': ObjectId('6118d887066a0b17c9a4a531'),
    'probes': [
        {
            'probe_id': ObjectId('6118d887066a0b17c9a4a530'),
            'readings': [
                {'value': 42, 'when': Date("2021-08-12T05:25:28.905Z")},
                {'value': 37, 'when': Date("2021-08-12T08:34:30.405Z")},
                {'value': 43, 'when': Date("2021-08-12T12:56:45.043Z")},
                ...
            ]
        },
        ...
    ]
}

Full specification

  1. Readings can be inserted into readings out of time order.
  2. readings must remain sorted by ascending when.
  3. There must never be 2 items in succession in readings with the same value.
  4. When a $push would violate rule 3 then the new item must replace the old if (and only if) it has an earlier when (if when is a lower value), otherwise the readings should remain untouched.
  5. The transaction must be atomic.

Bonus points

  1. If a probe subdocument with probe_id is not present in probes then create the {'probe_id':ObjectId(), 'readings':[]} subdocument during the $push.

Possible cases

  1. empty list - Add item
  2. push back, different value from preceding* member - Add item
  3. push back, same value as preceding member - Do nothing (don't push)
  4. push middle, different value from preceding and following** member - Add item
  5. push middle, same value as preceding member - Do nothing (don't push)
  6. push middle, same value as following member - Replace following member
  7. push front, different value from following member - Add item
  8. push front, same value as following member - Replace following member

*"preceding" = earlier in time (lower when)

**"following" = later in time (higher when)

Notes

  • Items with identical when may either be ignored or replaced or be pushed (provided value between all items with identical when are different)
  • From the point of optimization, we normally push back (insert with more recent, higher valued, when

Current Code

Using PyMongo:

result = collection.update_one(
    {'_id': location_id },
    {'$push': {'probes.$[probe].readings': {'$each': [new_reading], '$sort': {'when': 1}}}},
    array_filters = [{'probe.id': probe_id}],
)

if result.modified_count == 0:
    collection.update_one(
        {'_id': location_id },
        {'$push': {'probes': {'id': probe_id, 'readings': [new_reading]}}},
    )

Clearly this code does nothing to assert that consecutive items are of different values. It also fails to push the required subdocument if that is not present without 2 database calls. Is the above specification possible?

2 Answers

Solution 1

Online example for inserting {"value" : 43 ,"when" : 11}
Push middle, same value as following member - Replace following member
Test code here

All possible cases (its 9 cases, 7 + 2 add on empty probes("bonus" case) or only empty readings)

---------------------Empty probes(case 1(the extra case) empty probes)-------------------------------
Before add
{"_id": {"$oid": "61345b2aefdf45b6128444f2"}, "location_id": 1, "probes": []}
Case : Only 1 case exists Member : {value 41, when 3}
{"_id": {"$oid": "61345b2aefdf45b6128444f2"}, "location_id": 1, "probes": [{"probe_id": 4, "readings": [{"value": 41, "when": 3}]}]}

---------------------Empty readings(case 1 empty readings)-------------------------------
Before add
{"_id": {"$oid": "61345b68efdf45b612844650"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": []}]}
Case : Only 1 case exists Member : {value 41, when 3}
{"_id": {"$oid": "61345b68efdf45b612844650"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 41, "when": 3}]}]}

---------------------Not empty readings-------------------------------
Before add
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}
Case : NoConflict start[push front, different value from following member - Add item] Member : {value 41, when 3}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 41, "when": 3}, {"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}
Case : Conflict start[push front, same value as following member - Replace following member] Member : {value 42, when 3}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 3}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}
Case : NoConflict middle[push middle, different value from preceding and following** member - Add item] Member : {value 42, when 11}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 42, "when": 11}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}
Case : Conflict middle[push middle, same value as preceding member - Do nothing (don't push)] Member : {value 37, when 11}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}
Case : Conflict middle[push middle, same value as following member - Replace following member] Member : {value 43, when 11}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 11}, {"value": 41, "when": 20}]}]}
Case : NoConflict end[push back, different value from preceding* member - Add item] Member : {value 47, when 21}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}, {"value": 47, "when": 21}]}]}
Case : Conflict end[push back, same value as preceding member - Do nothing (don't push)] Member : {value 41, when 21}
{"_id": {"$oid": "61346a33efdf45b6128496b7"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}]}]}


Query

  • its big but its fast, its atomic, all done in 1 query, it doesn't sort the array,it inserts in the right position to keep it sorted

  • to move this code to your code replace the 4 with new ObjectID() (add the method call) replace the 1 with the location ObjectID replace the 2 with the prob_id ObjectID i used numbers instead of dates for easy testing, but no need to change the query for that

  • first $set is to define the new ObjectID(replace the 4 with the method call) in case probes is empty create the probe with empty readings

  • $map to get inside the array

  • $reduce to find the position where we will add the new member the when field defines the position, the query keep the array sorted by adding in the right place
    *we could use reduce to make the array not just find the position but if the array had >500 members because of concat it would be so slow not usable.

  • when we have the position we check 4 basic cases

    • add at empty readings (1 case)
    • add at start (2 cases)
    • add at middle (3 cases)
    • add at end (2 cases)
    • each one of them has other cases if Conflict of value or not
db.collection.update({
  "location_id": 1
},
[
  {
    "$set": {
      "prob-id": {
        "$cond": [
          {
            "$eq": [
              "$probes",
              []
            ]
          },
          4,
          2
        ]
      }
    }
  },
  {
    "$set": {
      "probes": {
        "$cond": [
          {
            "$eq": [
              "$probes",
              []
            ]
          },
          [
            {
              "probe_id": "$prob-id",
              "readings": []
            }
          ],
          "$probes"
        ]
      }
    }
  },
  {
    "$set": {
      "probes": {
        "$map": {
          "input": "$probes",
          "as": "m1",
          "in": {
            "$cond": [
              {
                "$ne": [
                  "$$m1.probe_id",
                  "$prob-id"
                ]
              },
              "$$m1",
              {
                "$mergeObjects": [
                  "$$m1",
                  {
                    "readings": {
                      "$let": {
                        "vars": {
                          "size_position": {
                            "$reduce": {
                              "input": "$$m1.readings",
                              "initialValue": [
                                0,
                                null,
                                null
                              ],
                              "in": {
                                "$let": {
                                  "vars": {
                                    "index_pos": "$$value",
                                    "m2": "$$this"
                                  },
                                  "in": {
                                    "$let": {
                                      "vars": {
                                        "index": {
                                          "$arrayElemAt": [
                                            "$$index_pos",
                                            0
                                          ]
                                        },
                                        "pos": {
                                          "$arrayElemAt": [
                                            "$$index_pos",
                                            1
                                          ]
                                        }
                                      },
                                      "in": {
                                        "$cond": [
                                          {
                                            "$and": [
                                              {
                                                "$eq": [
                                                  "$$pos",
                                                  null
                                                ]
                                              },
                                              {
                                                "$gt": [
                                                  "$$m2.when",
                                                  11
                                                ]
                                              }
                                            ]
                                          },
                                          [
                                            {
                                              "$add": [
                                                "$$index",
                                                1
                                              ]
                                            },
                                            "$$index"
                                          ],
                                          [
                                            {
                                              "$add": [
                                                "$$index",
                                                1
                                              ]
                                            },
                                            "$$pos"
                                          ]
                                        ]
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        },
                        "in": {
                          "$let": {
                            "vars": {
                              "asize": {
                                "$arrayElemAt": [
                                  "$$size_position",
                                  0
                                ]
                              },
                              "position": {
                                "$arrayElemAt": [
                                  "$$size_position",
                                  1
                                ]
                              }
                            },
                            "in": {
                              "$switch": {
                                "branches": [
                                  {
                                    "case": {
                                      "$eq": [
                                        "$$asize",
                                        0
                                      ]
                                    },
                                    "then": [
                                      {
                                        "value": 43,
                                        "when": 11
                                      }
                                    ]
                                  },
                                  {
                                    "case": {
                                      "$eq": [
                                        "$$position",
                                        null
                                      ]
                                    },
                                    "then": {
                                      "$let": {
                                        "vars": {
                                          "prv_member": {
                                            "$arrayElemAt": [
                                              "$$m1.readings",
                                              {
                                                "$subtract": [
                                                  "$$asize",
                                                  1
                                                ]
                                              }
                                            ]
                                          }
                                        },
                                        "in": {
                                          "$cond": [
                                            {
                                              "$eq": [
                                                "$$prv_member.value",
                                                43
                                              ]
                                            },
                                            "$$m1.readings",
                                            {
                                              "$concatArrays": [
                                                "$$m1.readings",
                                                [
                                                  {
                                                    "value": 43,
                                                    "when": 11
                                                  }
                                                ]
                                              ]
                                            }
                                          ]
                                        }
                                      }
                                    }
                                  },
                                  {
                                    "case": {
                                      "$eq": [
                                        "$$position",
                                        0
                                      ]
                                    },
                                    "then": {
                                      "$let": {
                                        "vars": {
                                          "next_member": {
                                            "$arrayElemAt": [
                                              "$$m1.readings",
                                              0
                                            ]
                                          }
                                        },
                                        "in": {
                                          "$cond": [
                                            {
                                              "$eq": [
                                                "$$next_member.value",
                                                43
                                              ]
                                            },
                                            {
                                              "$cond": [
                                                {
                                                  "$lt": [
                                                    11,
                                                    "$$next_member.when"
                                                  ]
                                                },
                                                {
                                                  "$concatArrays": [
                                                    [
                                                      {
                                                        "value": 43,
                                                        "when": 11
                                                      }
                                                    ],
                                                    {
                                                      "$slice": [
                                                        "$$m1.readings",
                                                        1,
                                                        "$$asize"
                                                      ]
                                                    }
                                                  ]
                                                },
                                                "$$m1.readings"
                                              ]
                                            },
                                            {
                                              "$concatArrays": [
                                                [
                                                  {
                                                    "value": 43,
                                                    "when": 11
                                                  }
                                                ],
                                                "$$m1.readings"
                                              ]
                                            }
                                          ]
                                        }
                                      }
                                    }
                                  }
                                ],
                                "default": {
                                  "$let": {
                                    "vars": {
                                      "next_member": {
                                        "$arrayElemAt": [
                                          "$$m1.readings",
                                          "$$position"
                                        ]
                                      },
                                      "prv_member": {
                                        "$arrayElemAt": [
                                          "$$m1.readings",
                                          {
                                            "$subtract": [
                                              "$$position",
                                              1
                                            ]
                                          }
                                        ]
                                      }
                                    },
                                    "in": {
                                      "$switch": {
                                        "branches": [
                                          {
                                            "case": {
                                              "$and": [
                                                {
                                                  "$ne": [
                                                    "$$next_member.value",
                                                    43
                                                  ]
                                                },
                                                {
                                                  "$ne": [
                                                    "$$prv_member.value",
                                                    43
                                                  ]
                                                }
                                              ]
                                            },
                                            "then": {
                                              "$concatArrays": [
                                                {
                                                  "$slice": [
                                                    "$$m1.readings",
                                                    0,
                                                    "$$position"
                                                  ]
                                                },
                                                [
                                                  {
                                                    "value": 43,
                                                    "when": 11
                                                  }
                                                ],
                                                {
                                                  "$slice": [
                                                    "$$m1.readings",
                                                    "$$position",
                                                    {
                                                      "$add": [
                                                        "$$asize",
                                                        1
                                                      ]
                                                    }
                                                  ]
                                                }
                                              ]
                                            }
                                          },
                                          {
                                            "case": {
                                              "$eq": [
                                                "$$prv_member.value",
                                                43
                                              ]
                                            },
                                            "then": "$$m1.readings"
                                          }
                                        ],
                                        "default": {
                                          "$concatArrays": [
                                            {
                                              "$slice": [
                                                "$$m1.readings",
                                                0,
                                                "$$position"
                                              ]
                                            },
                                            [
                                              {
                                                "value": 43,
                                                "when": 11
                                              }
                                            ],
                                            {
                                              "$slice": [
                                                "$$m1.readings",
                                                {
                                                  "$add": [
                                                    "$$position",
                                                    1
                                                  ]
                                                },
                                                {
                                                  "$add": [
                                                    "$$asize",
                                                    1
                                                  ]
                                                }
                                              ]
                                            }
                                          ]
                                        }
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                ]
              }
            ]
          }
        }
      }
    }
  },
  {
    "$unset": [
      "prob-id"
    ]
  }
])

Solution 2

Online example for {"probe_id" : 2, "readings" : [{"value" :43 ,"when" : 11}]}
Test code here

All possible cases (its 9 cases, 7 + 2 add on empty probes("bonus" case) or only empty readings)

---------------------Empty probes-------------------------------
Before add
{"_id": {"$oid": "61356ae9fdad8624e245e271"}, "location_id": 1, "probes": []}
Case : Only 1 case exists[empty list - Add item(the bonus case)] Member : {:probe_id 2, :readings [{value 42, when 3}]}
{"_id": {"$oid": "61356ae9fdad8624e245e279"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 3}], "probe_id": 2}]}

---------------------Empty readings-------------------------------
Before add
{"_id": {"$oid": "61356b73fdad8624e245e58c"}, "location_id": 1, "probes": [{"probe_id": 2, "readings": []}]}
Case : Only 1 case exists[empty list - Add item] Member : {:probe_id 2, :readings [{value 42, when 3}]}
{"_id": {"$oid": "61356b73fdad8624e245e594"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 3}], "probe_id": 2}]}

---------------------Not empty readings-------------------------------
Before add
{"_id": {"$oid": "613569f7fdad8624e245dd01"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : NoConflict start[push front, different value from following member - Add item] Member : {:probe_id 2, :readings [{value 41, when 3}]}
{"_id": {"$oid": "61356a2dfdad8624e245de34"}, "location_id": 1, "probes": [{"readings": [{"value": 41, "when": 3}, {"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : Conflict start[push front, same value as following member - Replace following member] Member : {:probe_id 2, :readings [{value 42, when 3}]}
{"_id": {"$oid": "61356a2dfdad8624e245de3d"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 3}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : NoConflict middle[push middle, different value from preceding and following** member - Add item] Member : {:probe_id 2, :readings [{value 42, when 11}]}
{"_id": {"$oid": "61356a2dfdad8624e245de46"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 42, "when": 11}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : Conflict middle[push middle, same value as preceding member - Do nothing (don't push)] Member : {:probe_id 2, :readings [{value 37, when 11}]}
{"_id": {"$oid": "61356a2efdad8624e245de4f"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : Conflict middle[push middle, same value as following member - Replace following member] Member : {:probe_id 2, :readings [{value 43, when 11}]}
{"_id": {"$oid": "61356a2efdad8624e245de59"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 11}, {"value": 41, "when": 20}], "probe_id": 2}]}
Case : NoConflict end[push back, different value from preceding* member - Add item] Member : {:probe_id 2, :readings [{value 47, when 21}]}
{"_id": {"$oid": "61356a2efdad8624e245de62"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}, {"value": 47, "when": 21}], "probe_id": 2}]}
Case : Conflict end[push back, same value as preceding member - Do nothing (don't push)] Member : {:probe_id 2, :readings [{value 41, when 21}]}
{"_id": {"$oid": "61356a2efdad8624e245de6b"}, "location_id": 1, "probes": [{"readings": [{"value": 42, "when": 5}, {"value": 37, "when": 10}, {"value": 43, "when": 15}, {"value": 41, "when": 20}], "probe_id": 2}]}

Query

  • uses another method, with lookup
  • adds the reading in the end, sorts the array by when
  • removes and element if the previous member has the same value (this way we keep only the members with the smaller when)
  • uses merge for the update, its not possible with update even with pipeline to do those, because we need lookup and group etc *merge requires a unique index on location_id else wont work *online example doesnt have the merge, query bellow has it

Drawback

  • if someone else add readings in the same location_id, while we update that location_id document (its very small thim) but stil if concurency of milliseconds is important it might cause data loss.

*You can use the logic of this query and parts of the code with transaction if you prefer smaller but more queries. For example add in the end, sort array (1 query), remove duplicate values (second query) etc, but you dont need solution2 or transactions, solution1 does all in 1 query + atomic but ok

db.testcoll.aggregate([
  {
    "$match": {
      "location_id": {
        "$eq": 1
      }
    }
  },
  {
    "$set": {
      "prob-id": {
        "$cond": [
          {
            "$eq": [
              "$probes",
              []
            ]
          },
          4,
          2
        ]
      }
    }
  },
  {
    "$set": {
      "probes": {
        "$cond": [
          {
            "$eq": [
              "$probes",
              []
            ]
          },
          [
            {
              "probe_id": "$prob-id",
              "readings": []
            }
          ],
          "$probes"
        ]
      }
    }
  },
  {
    "$lookup": {
      "from": "testcoll1",
      "let": {
        "probes": "$probes"
      },
      "pipeline": [
        {
          "$set": {
            "probes": {
              "$concatArrays": [
                "$$probes",
                [
                  {
                    "probe_id": 2,
                    "readings": [
                      {
                        "value": 43,
                        "when": 11
                      }
                    ]
                  }
                ]
              ]
            }
          }
        },
        {
          "$unwind": {
            "path": "$probes"
          }
        },
        {
          "$unwind": {
            "path": "$probes.readings"
          }
        },
        {
          "$sort": {
            "probes.probe_id": 1,
            "probes.readings.when": 1
          }
        },
        {
          "$replaceRoot": {
            "newRoot": "$probes"
          }
        }
      ],
      "as": "probes"
    }
  },
  {
    "$set": {
      "probes": {
        "$reduce": {
          "input": "$probes",
          "initialValue": [],
          "in": {
            "$let": {
              "vars": {
                "ps": "$$value",
                "p": "$$this"
              },
              "in": {
                "$let": {
                  "vars": {
                    "prv_p": {
                      "$last": "$$ps"
                    }
                  },
                  "in": {
                    "$cond": [
                      {
                        "$and": [
                          "$$prv_p",
                          {
                            "$eq": [
                              "$$prv_p.readings.value",
                              "$$p.readings.value"
                            ]
                          },
                          {
                            "$eq": [
                              "$$prv_p.probe_id",
                              "$$p.probe_id"
                            ]
                          },
                          {
                            "$eq": [
                              "$$p.probe_id",
                              2
                            ]
                          }
                        ]
                      },
                      "$$ps",
                      {
                        "$concatArrays": [
                          "$$ps",
                          [
                            "$$p"
                          ]
                        ]
                      }
                    ]
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  {
    "$lookup": {
      "from": "testcoll1",
      "let": {
        "probes": "$probes"
      },
      "pipeline": [
        {
          "$set": {
            "probes": "$$probes"
          }
        },
        {
          "$unwind": {
            "path": "$probes"
          }
        },
        {
          "$replaceRoot": {
            "newRoot": "$probes"
          }
        },
        {
          "$group": {
            "_id": "$probe_id",
            "readings": {
              "$push": "$readings"
            }
          }
        },
        {
          "$set": {
            "probe_id": "$_id"
          }
        },
        {
          "$project": {
            "_id": 0
          }
        }
      ],
      "as": "probes"
    }
  },
  {
    "$unset": [
      "_id",
      "prob-id"
    ]
  },
  {
    "$merge": {
        "into": {
          "db": "testdb",
          "coll": "testcoll"
        },
        "on": [
          "location_id"
        ],
        "whenMatched": "replace",
        "whenNotMatched": "discard"
    }
  }
])
Related