How Do I Fetch All Old Items on an RSS Feed?

user_14834 0
Viewed 74287

I've been experimenting with writing my own RSS reader. I can handle the "parse XML" bit. The thing I'm getting stuck on is "How do I fetch older posts?"

Most RSS feeds only list the 10-25 most recent items in their XML file. How do I get ALL the items in a feed, and not just the most recent ones?

The only solution I could find was using the "unofficial" Google Reader API, which would be something like

http://www.google.com/reader/atom/feed/http://fskrealityguide.blogspot.com/feeds/posts/default?n=1000

I don't want to make my application dependent on Google Reader.

Is there any better way? I noticed that on Blogger, I can do "?start-index=1&max-results=1000", and on WordPress I can do "?paged=5". Is there any general way to fetch an RSS feed so that it gives me everything, and not just the most recent items?

rss
7 Answers

RSS/Atom feeds does not allow for historic information to be retrieved. It is up to the publisher of the feed to provide it if they want such as in the blogger or wordpress examples you gave above.

The only reason that Google Reader has more information is that it remembered it from when it came up the first time.

There is some information on something like this talked about as an extension to the ATOM protocol, but I don't know if it is actually implemented anywhere.

user_678290

In my experience with RSS, the feed is compiled by the last X items where X is a variable. Certain Feeds may have the full list, but for bandwidth sake most places are likely limiting to just the last few items.

The likely answer for google reader having the old info, is that it is storing it on its side for users later.

Rob_Haupt_653880

Why does this problem exist?

Most RSS readers need to import feeds through a live URL, which makes things harder for sites that are unindexed on Wayback Machine.

The reason why Wayback Machine feeds can be imported is that the reader can regularly poll the server for updates according to its defined TTL configuration. The reader compares the current datetime with the RSS feed posts pubDate or lastBuildDate keys in the XML response. We can't hack the machine datetime to work around the datetime resolution because the current datetime is fetched live.

I've outlined an alternative solution without Wayback below. Unfortunately, I have not been able to find a universal solution for all feed sources.

Alternative Solution(s)

In my experience, NOT ALL feeds are partial though. The XML doesn't have to specify the datetime of each post. This means the RSS Reader doesn't have a datetime to filter the feed with. An example of this feed type can be found here.

This kind of reading experience is useful when chronological order is irrelevant, and the content doesn't need to be sorted. This approach is useful for sites where ALL the content is valuable, and the linked Essays of Paul Graham is a good example.

  1. If the site has a generic, non-chronological feed option, subscribe to that RSS instead (the preferred option).
  2. Download the linked timestamped .rss file, strip datetimes and host the file on your own server. Note, we can implement this via an AWS Lambda.
    1. Set up a server that fetches the RSS from live.
    2. Strip the pubDate tags from the XML file on fetch.
    3. Host the modified RSS on your own server.

Note

These are suboptimal solutions due to loss of orders, however, I wanted to provide a potential alternative to WaybackMachine.

In addition, some existing answers require advanced SysDesign workarounds, more prework and in some cases are outdated (Google Reader is shut down). I hope it's helpful for those who really need a solution for a complete feed list. Constructing new RSS feeds is not too hard from the original RSS file.

user_49958280
Related