Writing A Loop: Taking a List of URLS And Only Getting The Title Text and Meta Description - BeautifulSoup/Python

user_13170774 0
Viewed 207

I am a fairly new data worker in the public health field. Any help is appreciated.

Basically our goal is to create a quick way to extract the title and meta description from a list of URLs. We are using Python. We do not need anything else from the webpage.

I have the following list called "urlList". I have written out (using Beautiful Soup) the 

urlList  = https://www.freeclinics.com/cit/ca-los_angeles?sa=X&ved=2ahUKEwjew7SbgMXoAhUJZc0KHYHUB-oQ9QF6BAgIEAI,
https://www.freeclinics.com/cit/ca-los_angeles,
https://www.freeclinics.com/co/ca-los_angeles,
http://cretscmhd.psych.ucla.edu/healthfair/HF%20Services/LinkingPeopletoServices_CLinics_List_bySPA.pdf

Then I was able to extract the title and description of one of the URL's (see below code). I was hoping to loop this over the list. I am open to any form of data export - i.e. It can be a data table, .csv, or .txt file.

I know my current Print output shows the title and description as strings, where the description output is in [ ]. This is fine. My main concern of this post is looping through the whole urlList. 

urlList = "https://www.freeclinics.com/cit/ca-los_angeles?sa=X&ved=2ahUKEwjew7SbgMXoAhUJZc0KHYHUB-oQ9QF6BAgIEAI"

response = requests.get(urlList)
soup = BeautifulSoup(response.text)
metas = soup.find_all('meta')

print((soup.title.string),[ meta.attrs['content'] for meta in metas if 'name' in meta.attrs and meta.attrs['name'] == 'description' ])

>> Output: Free and Income Based Clinics Los Angeles CA ['Search below and find all of the free and income based health clinics in Los Angeles CA. We have listed out all of the Free Clinics listings in Los Angeles, CA below']

P.s - At the most, the urlList will have a max of 10-20 links. All are very similar in page structure.

web-scraping beautifulsoup python loops
1 Answers

You can define a function that takes urlList as an arguments and returns list of list where each sublist in main list contains title and its corresponding description.

Try this:

def extract_info(url_list):
    info = []
    for url in url_list:
        with requests.get(url) as response:
            soup = BeautifulSoup(response.text, "lxml")
            title = soup.find('title') .text if soup.find('title') else None
            description = soup.find('meta', {"name": "description"})["content"] if soup.find('meta', {"name": "description"}) else None
            info.append([title, description])
    return info

Output:

[['Free and Income Based Clinics Los Angeles CA',
  'Search below and find all of the free and income based health clinics in '
  'Los Angeles CA. We have listed out all of the Free Clinics listings in Los '
  'Angeles, CA below']
...
]]
user_128331660
Related