Get Feeds from FeedParser and Import to Pandas DataFrame

user_5155413 0
Viewed 3709

I'm learning python. As practice I'm building a rss scraper with feedparser putting the output into a pandas dataframe and trying to mine with NLTK...but I'm first getting a list of articles from multiple RSS feeds.

I used this post on how to pass multiple feeds and combined it with an answer I got previously to another question on how to get it into a Pandas dataframe.

What the problem is, I want to be able to see the data from all the feeds in my dataframe. Currently I'm only able to access the first item in the list of feeds.

FeedParser seems to be doing it's job but when putting it into the Pandas df it only seems to grab the first RSS in the list.

import feedparser
import pandas as pd

rawrss = [
    'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml',
    'https://www.yahoo.com/news/rss/',
    'http://www.huffingtonpost.co.uk/feeds/index.xml',
    'http://feeds.feedburner.com/TechCrunch/',
    ]

feeds = []
for url in rawrss:
    feeds.append(feedparser.parse(url))

for feed in feeds:
    for post in feed.entries:
        print(post.title, post.link, post.summary)

df = pd.DataFrame(columns=['title', 'link', 'summary'])

for i, post in enumerate(feed.entries):
    df.loc[i] =  post.title, post.link, post.summary

df.shape

df
pandas feedparser python
2 Answers

I'm using dict to build DataFrame:

import feedparser
import pandas as pd

rawrss = [
    'http://newsrss.bbc.co.uk/rss/newsonline_uk_edition/front_page/rss.xml',
    'https://www.yahoo.com/news/rss/',
    'http://www.huffingtonpost.co.uk/feeds/index.xml',
    'http://feeds.feedburner.com/TechCrunch/',
    ]

df = pd.DataFrame([])

for url in rawrss:
    dp = feedparser.parse(url)

    for i, e in enumerate(dp.entries):
        one_feed = {}
        one_feed['etitle'] = e.title if 'title' in e else f'title {i}'
        one_feed['summary'] = e.summary if 'summary' in e else f'no summary {i}'
        one_feed['elink'] = e.link if 'link' in e else f'link {i}'
        one_feed['published'] = e.published if 'published' in e else f'no published {i}'
        one_feed['elink_img'] = e.links[1].href if 'links' in e and len(e.links)>1 else f'no link_img {i}'

        df = df.append(pd.DataFrame([one_feed]), ignore_index=True)

It's easier to add columns this way.

user_170931080
Related