do I need to remove previously loaded videos to protect page performances?

On my page there are videos - and only videos
Because of performance issue - only a certain number of videos are loaded on page loading
the rest of them are loaded dynamically - let's say - like this:

$(window).scroll(function() {
    if($(window).scrollTop() == $(document).height() - $(window).height()) {
           // ajax call get data from server and append to the div
    }
});

but if I scroll enough - the page will also have too many videos - for example 1000 videos - isn't it ?

So the question is - do I need some code to remove the previously loaded videos which are not more in the viewport and load them again by scrolling up and down ?

it seems that executing that code for constantly load/remove videos - will also be a big burden for the page performances

any advice ?