I'm a newbie. As for my first project I'm trying to webscrape articles from google RSS and analyze them with NLTK. I've somehow managed to get my code working, but I'm struggling to clean the data.
Here's my code.
url = 'https://www.google.com/alerts/feeds/06940609537670719478/16876315740763892156'
resp = requests.get(url)
soup = BeautifulSoup(resp.content, "lxml")
items = soup.findAll('entry')
news_items = []
for entry in items:
news_item = {}
news_item['title'] = entry.title.text,
news_item['preview'] = entry.content.text,
news_item['published'] = entry.published.text,
news_item['updated'] = entry.updated.text,
news_item['link'] = entry.link['href']
news_items.append(news_item)
df = pd.DataFrame(news_items, columns=['title', 'preview', 'published', 'updated', 'link'])
df = df.sort_index()
My problem is that I can't do anything with the text. I can't make it all lower case, remove punctuation etc. As I checked data types of each column, they all are identified as objects with format of "%s" however pandas.covert_dtypes only changes last column's data type and even after the change it still won't let me clean it.
I was trying to use:
df['title'] = df['title'].str.lower()
or just convert 2 columns that consist of dates into datetime with specific format, but my code is just ignoring me.
Could you guys suggest what stops me from altering scraped text?
Moreover, as it is going to be a bigger project I was trying to put it into functions rather than let it just lie loosely, but I can't seem to get my code working in functions at all. My tries on that topic resulted in Code not giving me errors, but also not responding to calls and generally ignoring me.
If you could suggest me either a source or give me a tip on how to arrange the code so it would work within functions I would be super grateful.