Javascript prevent accidental modal cancel when the clicking is finished outside the modal

I have a modal dialog and a full screen background div that covers the screen. When I click outside the modal I would like to cancel/close it so I added an event listener to it. I also added an event listener to the modal to stop click propagation when the modal is clicked.

But when I select the text in the input field from right to left, like I start pressing the mouse button inside the input and release it when the mouse is on the left side of the input (over the background actually) then the "Outside click, cancel modal" is fired.

The same happens if the dialog has a scrollbar and I grab the scroll and when I release the mouse button outside the modal.

So I'd like "Outside click, cancel modal" event fired only if I really cliced the background, not any children of it.

How to prevent this accidental cancel?

document.getElementById("fullScreenModalBackground").addEventListener("click", (e) => {
  alert("Outside click, cancel modal");
});

document.getElementById("modal").addEventListener("click", (e) => {
  e.stopPropagation();
});

document.getElementById("btnClose").addEventListener("click", (e) => {
  alert("Close modal");
});

#fullScreenModalBackground {
  width: 200px;
  height: 200px;
  background-color: azure;
}

#modal {
  position: absolute;
  left: 50px;
  top: 50px;
  width: 100px;
  height: 100px;
  background-color: beige;
}

#modal input {
  width: 50px
}

<div id="fullScreenModalBackground">
  <div id="modal">
    <input value="lorem" />
    <button id="btnClose">Close modal</button>
  </div>
</div>