Use browser history state with next js

I am trying to implement custom scrolling with nextjs. I want to be able to save the scroll position of the page when a user leaves so when they return with popstate I can reset the scroll position.

The issue is that when I try to use replaceState with the history api next/router overrides what I put there with their own data. Next/router doesn't provide any type of history for their router and they don't put an id or anything that I can reference to use my own state. So I have no way to reference what page of the history stack I'm actually on. Next/router api docs are located here Nextjs Router API.

Here is a quick example of what I'm trying to do:

const setHistory = () => {
    const { state } = window.history;
    const newState = {
        ...state,
        scroll: 'my custom scroll value'
    };
    history.replaceState(newState, '', state.url);
}

const handleRouteChangeStart = (url) => {
    setHistory();
}

const handlePopstate = () => {
    console.log(window.history)
}

window.addEventListener('popstate', handlePopstate);
Router.events.on('routeChangeStart', handleRouteChangeStart);

I have also tried to set the value to the options key that nextjs sets but I have inconsistent results with that as well meaning sometimes the value isn't set and it feels a little hacky.

Does NextJs really not allow you to have any kind of interaction with browser history or the ability to manipulate the window.history object?