How to use 'keydown ' eventHandler with 'IntersectionObserver' in JavaScript?

I am configuring a slider in JavaScript. For the same I have written following code. slides is a NodeList of all 4 photos. sliderBtnLeft is a left arrow sliderBtnRight is a right arrow

there is a goToSlide(slide) function that is used to display the particular slide by adjusting the transformX.

there are 2 functions nextSlideFn() and prevSlideFn() that are called when corresponding buttons are pressed.

The code is working fine when I use buttons. The code also works okay when I use 'keydown' eventListener.

Problem arises when I try to use 'keydown' eventListener. with the intersection observer.

It works okay the first time section comes in the viewport. Later on if I scroll up and scroll down again, the function after ArrowKey is pressed gets called 3 times.

Eg. If i press ArrowRight after scrolling up and down, nextSlideFn() gets called 3 times. And the number of times it gets called increases by 2 everytime I scroll up and down.

One up and down scroll: nextSlideFn() gets called 3 times Twice up and down scroll: nextSlideFn() gets called 5 times Thrice up and down scroll: nextSlideFn() gets called 7 times and so on...

Same for ArrowLeft. (I checked it by logging it to the console)

There is some problem with the observer function sliderFunction() and document.addEventListener('keydown',...) when used together.
What might be the reason

//Code snippet

    const slides = document.querySelectorAll('.slide');
    const slider = document.querySelector('.slider');
    const sliderBtnLeft = document.querySelector('.slider__btn--left');
    const sliderBtnRight = document.querySelector('.slider__btn--right');
    
    let currentSlide = 0;
          
    
    const goToSlide = function(slide){
      slides.forEach((s, index) => {
        s.style.transform = `translateX(${100 * (index - slide)}%)`;
      });
    }
    
    goToSlide(0);
    
    const nextSlideFn = function(){
      (currentSlide === slides.length - 1) ? currentSlide = 0 : currentSlide++;
      console.log('Right pressed');
      goToSlide(currentSlide);
    };
    
    const prevSlideFn = function(){
      (currentSlide === 0) ? currentSlide = slides.length - 1 : currentSlide --;
      console.log('Left pressed');
      goToSlide(currentSlide);
    };
    
    sliderBtnRight.addEventListener('click', nextSlideFn);
    sliderBtnLeft.addEventListener('click', prevSlideFn);
    
    const slideObject = {
      root: null,
      threshold: 0,
    }
    
    const sliderFunction = function(entries,observer){
      const entry = entries[0];
      console.log(entry.target);
      
      document.addEventListener('keydown', function(e){
    
        if(e.key === 'ArrowRight'){
          nextSlideFn();
        }
        if(e.key === 'ArrowLeft'){
           prevSlideFn();
        }
       }); 
   
    };
    
    const slideObserver = new IntersectionObserver(sliderFunction, slideObject);

    slideObserver.observe(slider);