Assigning different streams to `srcObject` of a video element

I have a video element. I have multiple streams captured by navigator.getUserMedia.

I can assign srcObject successfully the first time:

previewVideoElement.srcObject = stream;

However if I re-assign a different stream to srcObject later (same element) then the stream doesn't work (no errors, blank video). How can I do this without recreating video elements each time?

Edit: trying this fails as well:

const previewVideoElement = document.getElementById("new-device-preview");
previewVideoElement.pause();
previewVideoElement.srcObject = stream;
previewVideoElement.play();

Edit: calling this works a few times, but then fails with The play() request was interrupted by a call to pause(). Without pause I get The play() request was interrupted by a new load request..

previewVideoElement.pause();
previewVideoElement.srcObject = stream;
previewVideoElement.load();
previewVideoElement.play();

Edit: even this heap of garbage doesn't work:

const previewVideoElement = document.getElementById("new-device-preview");
//previewVideoElement.pause();
previewVideoElement.srcObject = stream;
previewVideoElement.load();
const isPlaying = previewVideoElement.currentTime > 0 && !previewVideoElement.paused && !previewVideoElement.ended && previewVideoElement.readyState > 2;
if (!isPlaying)
    setTimeout(function () {
         previewVideoElement.play();
    }, 500);

The only thing I could get working reliably:

var previewVideoElement = document.getElementById("new-device-preview");
if (previewVideoElement.srcObject) {
      $("#new-device-preview-container").empty();
      $("#new-device-preview-container").html('<video autoplay class="new-device-preview" id="new-device-preview"></video>')
}
previewVideoElement = document.getElementById("new-device-preview");
previewVideoElement.srcObject = stream;