Keep array sorted after update in MongoDB

Suppose I have a collection that looks like this:

[
   {
      "key":1,
      "xs":[
         {
            "x":1,
            "y":2
         },
         {
            "x":3,
            "y":4
         }
      ]
   },
   {
      "key":2,
      "xs":[
         {
            "x":5,
            "y":6
         },
         {
            "x":7,
            "y":8
         }
      ]
   }
]

The schema is not exactly relevant, but the main idea is that I want to keep the xs array sorted by the x field, after potentially adding, removing or updating objects within xs.

Now adding/deleting are straightforward, but I'm using $set for field-wise updates, and I'm having issues following it up with a

$push: {
  xs: {
    $each: [],
    $sort: { x: 1 }
  }
}

Mongoose complains with a Updadting the path ... would create a conflict at 'x' error, and I'm not exactly sure how to avoid it.