How do I scrape the word "Kentucky" from the whole page?

user_19401467 0
Viewed 45

Every time I run this code it only gives me the data three times but I know it pops up way more times on the page.

   from bs4 import BeautifulSoup
   import requests

   url = 'https://www.nba.com/players'
   result = requests.get(url)
   doc = BeautifulSoup(result.text, 'lxml')
   college = doc.find_all(text='Kentucky')
   print(college)
web-scraping python-3.x
1 Answers

It is because find_all with text will search for exact text match.

college = doc.find_all(text='Kentucky')

Valid one: <option value="Kentucky">Kentucky</option>.
No valid: <option value="Western Kentucky">Western Kentucky</option>.

If you want to find all occurrences you need to use regular expressions.
Check beautiful-soup-doc. Also use string instead of text.

The string argument is new in Beautiful Soup 4.4.0.
In earlier versions it was called text:

import re
from bs4 import BeautifulSoup
import requests

url = 'https://www.nba.com/players'
result = requests.get(url)
doc = BeautifulSoup(result.text, 'lxml')
college = doc.find_all(string=re.compile("Kentucky"))
print("Total elements: " + str(len(college)))
print(college)

OUTPUT:

Total elements: 6
['Kentucky', 'Western Kentucky', 'Kentucky', 'Western Kentucky', 'Kentucky', '{"props":{"pageProps":{"la
user_152135030
Related