Get the count of a phrase in a url using python and bs4

user_10487244 0
Viewed 43

I want to get the count of any phrase appearing in a URL, say https://en.wikipedia.org/wiki/India.

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/wiki/India'

r = requests.get(url)
soup = BeautifulSoup(r.text,'lxml')

Now, I want to get the count of the phrase India is a in the soup. How to go about this?

Please suggest.

web-scraping beautifulsoup python-3.x
2 Answers

This can be done in one of two ways.

First, the common denominator:

texts = soup.find_all(text=True)
cleaned = ["".join(t.strip()) for t in texts]
counter=0

Now, if you want to use regex:

import re
regex = re.compile(r'\bIndia is a\b') 
for c in cleaned:
    if regex.match(c) is not None:
        counter+=1

I, personally, don't like using regex except as last resort, so I would go the longer way

phrase = 'India is a'
for c in cleaned:   
            if phrase==c or phrase+' ' in c: 
                counter+=1

In both cases, print(counter) outputs 6.

Note that, intentionally, these do not count the 3 situations where the phrase is part of a larger phrase (such as India is also); it counts only the exact phrase or the phrase followed by a space.

user_94480900

I tried below and the same worked fine:

import re
import requests
url = 'https://en.wikipedia.org/wiki/India'

response = requests.get(url)
response_text = response.text
keyword = 'India is a'

match = re.findall("%s" % keyword, response_text)
count = (len(match))
count

Output is 9.

This code will look into <head>, <body> and elsewhere.

user_104872440
Related