Python: counting specific words from HTML

user_13627352 0
Viewed 284

So, I'm a Python rookie having quite a hard time learning web scraping. I intend on counting the number of words in this HTML page, and show which words appear only once, as well as how many times the word "ladies" is shown. I've managed to come up with this so far:

import requests
from bs4 import BeautifulSoup
import operator
from collections import Counter

def my_start(url):
   my_wordlist = []
   my_source_code = requests.get(url).text
   my_soup = BeautifulSoup(my_source_code, 'html.parser')
   for each_text in my_soup.findAll('p', {'class':'about-text'}):
      content = each_text.text
      words = content.lower().split()
      for each_word in words:
         my_wordlist.append(each_word)
      clean_wordlist(my_wordlist)

def clean_wordlist(wordlist):
   clean_list =[]
   for word in wordlist: 
      symbols = '!@#$%^&*()_-+={[}]|\;:"<>?/., '
      for i in range (0, len(symbols)):
         word = word.replace(symbols[i], '')
      if len(word) > 0:
         clean_list.append(word)
   create_dictionary(clean_list)

def create_dictionary(clean_list):
   word_count = {}
   for word in clean_list:
      if word in word_count:
         word_count[word] += 1
      else:
         word_count[word] = 1
   c = Counter(word_count)
   print(c)
   if word_count[word] == 1:
    print(word)
   top = soup.find_all("ladies")
   print(top)

if __name__ == '__main__':
  my_start("http://brasil.pyladies.com/about/")

I noticed that there are some words that appear only once that aren't shown here, as well as a word that appears twice and is shown. I can't figure out how to count the number of times the word "ladies" is shown either. Any inputs on the matter will be greatly appreciated!

python-requests beautifulsoup python
2 Answers

I recommend you to use regular expressions (regex) for solving this problem

import re

my_source_code = requests.get(url).text
pattern = "ladies"

ladies_count = len(re.findall(pattern, my_source_code))

it is most fast way for counting word from text

user_127551870

top = soup.find_all("ladies")

Here the usage of find_all is wrong. It is used to search for HTML tags, not words.

If you want to print the number of times the word 'ladies' has appeared, try

print(word_count.get('ladies','0'))
user_137500060
Related