What is the return type of the find_all method in Beautiful Soup?

user_12939884 0
Viewed 3423
from bs4 import BeautifulSoup, SoupStrainer 
from urllib.request import urlopen
import pandas as pd 
import numpy as np 
import re
import csv
import ssl
import json
from googlesearch import search
from queue import Queue
import re 

links = []
menu = []
filtered_menu = []


def contains(substring, string):
     if substring.lower() in string.lower():
         return True
     else:
         return False


for website in search("mr puffs", tld="com", num=1, stop=1, country="canada", pause=4): 
 links.append(website)


soup = BeautifulSoup(urlopen(links.pop(0)), features="html.parser")
menu = soup.find_all('a', href=True)

for string in menu:
    if contains("contact", string):
      filtered_menu.append(string)


print(filtered_menu)

I am creating a webscraper that will extract contact information from sites. However, in order to do that, I need to get to the contact page of the website. Using the googlesearch library, the code searches for a keyword and puts all the results (up to a certain limit) in a list. For simplicity, in this code, we are just putting in the first link. Now, from this link, I am creating a beautiful soup object and I am extracting all the other links on the website(because the contact information is usually not found on the homepage). I am putting these links in a list called menu.

Now, I want to filter menu for only links that have "contact" in it. Example: "www.smallBusiness.com/our-services" would be deleted from the new list while "www.smallBusiness.com/contact" or "www.smallBusiness.com/contact-us" will stay in the list.

I defined a method that checks if a substring is in a string. However, I get the following exception:

TypeError: 'NoneType' object is not callable.

I've tried using regex by doing re.search but it says that the expected type of string or byte-like value is not in the parameters.

I think it's because the return type of find_all is not a string. It's probably something else which I can't find in the docs. If so, how do I convert it into a string?

As requested in the answer below, here's what printing menu list gives:

From here, I just want to extract the highlighted links:

here is the image

web-scraping beautifulsoup python regex
2 Answers

BeautifulSoup.find_all() type is bs4.element.ResultSet (which is actually a list)

Individual items of find_all(), in your case the variable you call "string" are of type bs4.element.Tag.

As your contains function expects type str, your for loop should look something like:

for string in menu:
    if contains("contact", str(string)):
      filtered_menu.append(string)
user_121495870

This is how I did it to search google for user-inputted search terms, then scrape all the urls from the serps. The program goes on to visit each of those links directly and scrape text off of those.

Maybe you could modify this for your purposes?

 #First-stage scrape of Google

    headers = {"user-agent": USER_AGENT}
    URL=f"https://google.com/search?q={squery}"
    URL2=f"https://google.com/search?q={squery2}"
    URL3=f"https://google.com/search?q={squery3}"
    URL4=f"https://google.com/search?q={squery4}"
    resp=requests.get(URL, headers=headers)
    resp2=requests.get(URL2, headers=headers)
    resp3=requests.get(URL3, headers=headers)
    resp4=requests.get(URL4, headers=headers)

    results=[]
    s2results=[]
    s3results=[]
    s4results=[]
    def scrapeURL(a,b,c):
        if a.status_code == 200:
            print("Searching Google for information about: ", c)
            soup = BeautifulSoup(a.content, "html.parser")
            for g in soup.find_all('div', class_='r'):
                anchors = g.find_all('a')
                if anchors:
                    link = anchors[0]['href']
                    title = g.find('h3').text
                    item = {
                    link
                    }
                    b.append(item)
                else:
                    print("Couldn't scrape URLS from first phase")
        else: 
            print("Could not perform search. Status code: ",a.status_code)
    #Create list of urls and format to enable second scrape


    scrapeURL(resp,results,query)
    scrapeURL(resp2,s2results,query2)
    scrapeURL(resp3,s3results,query3)
    scrapeURL(resp4,s4results,query4)
    #Create list of urls and format to enable second scrape
    def formaturls(res,resstorage):
        a=0
        listurls=str(res)
        listurls=listurls.replace("[","")
        listurls=listurls.replace("{","")
        listurls=listurls.replace("'}","")
        listurls=listurls.replace("]","")
        listurls=listurls.lower()
        re=listurls.split(",")
        for items in re:
            s=str(re[a])
            resstorage.append(s)
            a=a+1 
    qresults=[]
    q2results=[]
    q3results=[]
    q4results=[]
    formaturls(results,qresults)
    formaturls(s2results,q2results)
    formaturls(s3results,q3results)
    formaturls(s4results,q4results)
user_127711240
Related