Python BeautifulSoup - extract URLs and request pages and then retrieve abstracts

user_16613055 0
Viewed 176

I want to access the E-journal page and then retrieve every abstract of the articles. So I wrote the code that makes a list of the URLs of abstract pages. And it works successfully. But when I tried to request the URLs and retrieve the abstracts, it didn't work. (with many 'None' in the console.)

This is my code.

import requests
from bs4 import BeautifulSoup

h = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36'}

URL = "https://agsjournals.onlinelibrary.wiley.com/toc/15325415/2021/69/7"

JAGS_result = requests.get(URL, headers=h)
JAGS_soup = BeautifulSoup(JAGS_result.text, "html.parser")

L = []
for link in JAGS_soup.find_all('a',{"title":"Abstract"}): 
   L.append(link.get('href')) 


Ab_Links = []
a = 0 
for ab_link in L:
  if a == len(L):
    break
  else:
    full_link = "https://agsjournals.onlinelibrary.wiley.com"+L[a]
    Ab_Links.append(full_link)
    a = a+1
print(Ab_Links)


b = 0
Ab = []
Ab_URL = Ab_Links[b]
for ab_url in Ab_Links:
  if b == len(L):
    break
  else:
   Ab_result = requests.get(Ab_Links[b], headers = h)
   Ab_soup = BeautifulSoup(Ab_result.text, "html.parser")
   abstract = Ab_soup.find({"class" : "article-section article-section__abstract"})
   Ab.append(abstract)
   b = b+1
print(Ab)

I am a novice to python and HTML so it is very hard to write code by myself. Please help me...

beautifulsoup python
3 Answers
import requests
from bs4 import BeautifulSoup, SoupStrainer
from urllib.parse import urljoin

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/90.0'
}


def main(url):
    with requests.Session() as req:
        req.headers.update(headers)
        r = req.get(url)
        soup = BeautifulSoup(
            r.content, 'lxml', parse_only=SoupStrainer('a', title='Abstract'))
        links = [urljoin(url, x['href']) for x in soup.select('a')]

        for link in links:
            r = req.get(link)
            soup = BeautifulSoup(r.text, 'lxml')
            print(soup.select_one('.article-section.article-section__abstract'))


if __name__ == "__main__":
    main('https://agsjournals.onlinelibrary.wiley.com/toc/15325415/2021/69/7')
user_76589850

Your code is mostly correct. The problem is with finding the abstract. In order to search for an element by class, use class_='...'. If you change your Abstract = line to the following, it will return results:

abstract = Ab_soup.find(class_='article-section article-section__abstract')

Also, you can simplify your loops. for ab_link in L will iterate through each item in L and then stop. You do not need to test if a == len(L), and in fact that code will never be True, because the loop will exit before a == len(L).

user_13422570

You could try this out.

This prints the abstract of all the articles in the page.

import requests
import bs4 as bs
url = 'https://agsjournals.onlinelibrary.wiley.com/toc/15325415/2021/69/7'
h = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36'}
resp = requests.get(url, headers=h)
soup = bs.BeautifulSoup(resp.text, 'lxml')

base_url = 'https://agsjournals.onlinelibrary.wiley.com'
abstract_urls = soup.findAll('a', attrs= {'title': 'Abstract'})
for i in abstract_urls:
    a_url = base_url + i['href']
    r = requests.get(a_url,headers=h)
    soup = bs.BeautifulSoup(r.text, 'lxml')
    abs_text = soup.find('section', class_='article-section article-section__full').text.strip()
    print(abs_text)
user_27732060
Related