Awkward problem with iterrating over the list and extracting only last linked link from the page [BS4]

user_13264096 0
Viewed 49

I am trying to scrape the website, and there are 12 pages with X links on them - I just want to extract all the links, and store them for later usage.

But there is an awkward problem with extracting links from the pages. To be precise, my output contains only the last link from each of the pages.

I know that this description may sound confusing, so let me show you the code and images:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv
import time

#here I tried to make a loop for generating page's URLs, and store URLs in the list "issues" 

archive = '[redacted URL]'
issues =[]
#i am going for issues 163-175
for i in range(163,175): 
    url_of_issue = archive + '/' + str(i)
    issues.append(url_of_issue)

#now, I want to extract links from generated pages
#idea is simple - loop iterates over the list of URLs/pages and from each issue page get URLS of the listed papers, storing them in the list "paper_urls" 

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
    paper_urls.append(ahrefTags)
    print(paper_urls)
    time.sleep(5)

But problem is, my output looks like [redacted]. Instead of ~80 links, I'm getting this! I wondered what happened, and it looks like my script from every generated URL (from the list named "issues" in the code) gets only the last listed link?! How to fix it? I do not have any idea what should be the problem here.

web-scraping beautifulsoup python-3.x python
1 Answers

Were you perhaps missing an indentation when appending to paper_urls?

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
        paper_urls.append(ahrefTags)    # added missing indentation
    print(paper_urls)
    time.sleep(5)

The whole code, after moving the print outside the loop, would look like this:

import requests
from bs4 import BeautifulSoup
import pandas as pd
import csv
import time

#here I tried to make a loop for generating page's URLs, and store URLs in the list "issues" 

archive = '[redacted URL]'
issues =[]
#i am going for issues 163-175
for i in range(163,175): 
    url_of_issue = archive + '/' + str(i)
    issues.append(url_of_issue)

#now, I want to extract links from generated pages
#idea is simple - loop iterates over the list of URLs/pages and from each issue page get URLS of the listed papers, storing them in the list "paper_urls" 

paper_urls =[]

for url in issues: 
    response = requests.get(url)
    html = response.text
    soup = BeautifulSoup(html, "html.parser")
    for a in soup.select('.obj_article_summary  .title a'):
        ahrefTags=(a['href'])
        paper_urls.append(ahrefTags)
        #print(ahrefTags)   #uncomment if you wish to print each and every link by itself
    #time.sleep(5)    #uncomment if you wish to add a delay between each request
print(paper_urls)
user_154016100
Related