Make a Div to return to its original position when scrolled up

I have a div with animation when you scrolls down and up, the problem is that when I scroll up and down very fast without letting the div to complete its animation the div little by little gets out of the screen in the upper part.

If I remove the .stop() in the .animate() function and scroll up and down very fast, the div keeps doing this for a while.

I want to keep the animation when scrolled up and down with the only difference that the menu does not get out of the screen when scrolling up and down fast, I have look thorough stack overflow questions like this one but nothing that I found work the code of the jsfiddle can be found here:

http://jsfiddle.net/QLLkL/4/

$(window).scroll(function(){
    if($(window).scrollTop() > 480 && !animationStarted){
         $("#menu").stop().animate({ "top": "+=180px" }, 1000);
        animationStarted  = true;
    };
    if($(window).scrollTop() < 480 && animationStarted){
         $("#menu").stop().animate({ "top": "-=180px" }, 1000);          
        animationStarted  = false;
    };        
});