Python Html: Extract Parts of Text from html file

user_4393903 0
Viewed 239

I'm currently working on a project where I downloaded a bunch of related html files and I gather data from them. One thing I noticed is that even though the overall format of the html files are similar, sometimes various files use different tags for storing similar information.

For example, in one file it could be:

<html>
<head>
<p> Title: The GodFather </p>
<p> Author: Mario Puzo </p>
</head>
<html>

And in another example it could be:

<html>
<head>
<p> Heading </p>
<pre> Ebook from xyz site: Please donate to our foundation at www.abc.com
Title: The GodFather
Author: Mario Puzo
</pre>
</head>
</html>

I can say for sure that "Title: " and "Author: " are common in all the html files. I want to extract the text that is next to "Title: " and "Author: ". I'm supposing I use beautiful soup to extract each html file. But to extract Title and Author, would using regular expressions be best?

natural-language-processing web-scraping beautifulsoup html python
1 Answers

Don't even bother with beautiful soup, just use regular expression:

re.findall(r'(?<=Author:).*?(?=<)', html.replace('\n', ''))
>>> [' Mario Puzo']

re.findall(r'(?<=Title:).*?(?=<)', html.replace('\n', ''))
>>> [' The GodFatherAuthor: Mario Puzo']

This will match Authors, granted, it may not work for some of your title because it also contained 'Author' before another tag (shown above), in which case you can do title.split('Author')[0] for all title crawled, since if Author is not in the string this method will not change the string.

user_80462320
Related