How to prevent duplicates at algorithm-based infinite scrolling

user_12381035 0
Viewed 304

This was more of a shower thought. So, nearly any social media platform has infinite scrolling. But how do they prevent duplicates?

So, lets say you fetch content with ajax. When, for example, you have a static list of 50 items, and you want to fetch 10 more results everytime the user has reached the bottom, you can easily sort by e.g. page:

$.ajax(
   url = "mysite.api/content?page=2"
   ...
)

But for a website like YouTube, your recommendations change probably every second and results cannot be sorted by page. E.g. if I wanted to fetch content for YouTube startpage it'd look something like

$.ajax(
   url = "youtube.api/start?user=aXqq1...&page=1"
   ...
)

But as soon as I scroll further, and YouTube fetches another page, the algorithm might already recommend different videos, or resort some of them, so duplicates might appear. So that's not an option, and I'd really like how they prevent it.

My current thoughts were:

1. Websockets:

YouTube queries for video content once and then loads more from the preloaded list everytime i scroll to the bottom.

Problem: YouTube doesn't know how far I will scroll. That would mean either they store a ton of video recommendations or they execute a new query after, lets say, 500 videos, which would again lead to the problem of duplicates.

2. Including already listed videos in ajax request

Everytime new videos are fetched, YouTube posts the list of videos (which are already loaded) back to the api so it can exclude those.

Problem: The loading times become slower every time a new request is made as more data has to be exchanged.

Which gets me right to where I started: How do they (efficiently) prevent duplicates?

infinite-scroll youtube algorithm ajax
1 Answers

Even on paginated results, you can run into the same problem. I've been on sites where when I click to the second page, it has results that were also on the first page (StackOverflow is one). I think it's a decision that trades off computational efficiency against quality of results. A simple solution that I found here says that you can accomplish it by just keeping a timestamp of when the user started looking, and then filtering results to exclude anything newer than that time. That should be relatively cheap to implement, but it means that you need to reload to get more relevant information. It wouldn't surprise me if this is the standard way of doing it, given how much apps and websites build refreshing into the workflow. For example, refreshing reddit is frustratingly easy to do by accident, and the youtube app refreshes its reccomendations when you switch away from it for a few minutes and come back. If users have come to expect that the need to refresh to get the latest results (which in my opinion, they have), then this kind of arguably sub-optimal solution is probably the way to go.

user_93079420
Related