CombineLatest of dynamic array inside switchMap is unsubscribing and re-subscribing continuously

I have at least two buttons that I want to dynamically listen for clicks on. listeningArray$ will emit an array (ar) of button #'s that I need to be listening to. When somebody clicks on one of these buttons I'm listening to, I need to console log that the button that was clicked and also log the value from a time interval.

If ar goes from [1,2] to [1], we need to stop listening to clicks on button #2. So the DOM click event needs to be removed for 2 and that should trigger the .finally() operator. But for 1, we should remain subscribed and the code inside the .finally() should not run, since nothing is being unsubscribed.

const obj$ = {};

Rx.Observable.combineLatest(
  Rx.Observable.interval(2000),
  listeningArray$ // Will randomly emit either [1] or [1,2]
)
.switchMap(([x, ar]) => {
    const observables = [];

    ar.forEach(n => {
        let nEl = document.getElementById('el'+n);

        obj$[n] = obj$[n] || Rx.Observable.fromEvent(nEl, 'click')
          .map(()=>{
            console.log(' el' + n);
          })
          .finally(() => {
            console.log(' FINALLY_' + n);
          });

        observables.push(obj$[n]);
    })

    return Rx.Observable.combineLatest(...observables);
})
.subscribe()

But what's happening is every time the interval emits a value, the DOM events ALL get removed and then immediately get added on again, and the code inside the .finally operator runs for 1 and 2.

This is really frustrating me. What am I missing?

It's a bit of a complex situation, so I created this: https://jsfiddle.net/mfp22/xtca98vx/7/