I am trying to scrape the title, summary, date, and link from http://www.indiainfoline.com/top-news for each div. with class' : 'row'.
link = 'http://www.indiainfoline.com/top-news'
redditFile = urllib2.urlopen(link)
redditHtml = redditFile.read()
redditFile.close()
soup = BeautifulSoup(redditHtml, "lxml")
productDivs = soup.findAll('div', attrs={'class' : 'row'})
for div in productDivs:
result = {}
try:
import pdb
#pdb.set_trace()
heading = div.find('p', attrs={'class': 'heading fs20e robo_slab mb10'}).get_text()
title = heading.get_text()
article_link = "http://www.indiainfoline.com"+heading.find('a')['href']
summary = div.find('p')
But none of the components are getting fetched. Any suggestion on how to fix this?