Weird scroll behavior when with chrome and smooth scrolling

I'm writing a virtual scrolling lib and encounter a weird problem which only appears in chrome and disable smooth scrolling in chrome flag fix it. One part of the logic is to use an invisible div to stretch the container and when scroll to top, fetch some new data then scroll back to old top. However, in chrome with smooth scrolling, it seems that changes to scrollTop will not be applied immediately so multiple data is fetched and scroll position is wrong.

I'm using chrome 84.0.4147.105-1 on Linux and have tested on Windows with same result. It may sometimes be hard to trigger, the safest way is to drag the scrollbar near top without trigger fetching and scroll up. Then you shall find console value jump from 0 to 900(scroll down) to a small number(why?) and trigger fetch again

const container = document.getElementById('container')
const scroller = document.getElementById('scroller')

let height = 0
let isFetching = false
const fetchData = h => {
    if (isFetching) return
    isFetching = true
    setTimeout(() => {
        height += h
        scroller.style.height = height + 'px'
        container.scrollTop += h
        isFetching = false
    }, 50)
}

fetchData(800)

container.addEventListener('scroll', ev => {
    const top = ev.target.scrollTop
    console.log(top)
    if (top <= 50) {
        fetchData(800)
    }
})

#container {
    height: 200px;
    width: 500px;
    background-color: blue;
    overflow: auto;
}

#scroller {
    width: 100%;
}

<div id="container">
     <div id="scroller"></div>
</div>