Loop through Hrefs in Python Using BeautifulSoup

user_13470631 0
Viewed 38

So I am trying to extract all the article URLs from this link.

However I only get ['https://mn.usembassy.gov/mn/2020-naadam-mn/', 'https://mn.usembassy.gov/mn/06272020-presidential-proclamation-mn/', 'https://mn.usembassy.gov/mn/pr-060320-mn/', 'https://mn.usembassy.gov/mn/dv-2021-status-check-mn/', 'https://mn.usembassy.gov/mn/pr-050120-mn/']

below is my code currently.

The website has 52 pages and I am trying to get all the URLs, why does it only give me a few URLs instead of all the URLS?

import requests
from bs4 import BeautifulSoup
url = 'https://mn.usembassy.gov/mn/news-events-mn/'
reqs = requests.get(url)
soup = BeautifulSoup(reqs.text, 'lxml')

urls = []
for h in soup.find_all('h2'):
    a = h.find('a')
    urls.append(a.attrs['href'])
print(urls)
beautifulsoup python-3.x python
1 Answers

The page contains only 5 links to articles, you need to go to next page to load next 5 links. This script will get all links from the page:

import requests
from bs4 import BeautifulSoup


url = 'https://mn.usembassy.gov/mn/news-events-mn/page/{page}/'

urls = []
for page in range(1, 53):
    soup = BeautifulSoup(requests.get(url.format(page=page)).content, 'html.parser')
    for h in soup.find_all('h2'):
        a = h.find('a')
        print(a['href'])
        urls.append(a.attrs['href'])


from pprint import pprint
pprint(urls)

Prints:

https://mn.usembassy.gov/mn/2020-naadam-mn/
https://mn.usembassy.gov/mn/06272020-presidential-proclamation-mn/
https://mn.usembassy.gov/mn/pr-060320-mn/
https://mn.usembassy.gov/mn/dv-2021-status-check-mn/
https://mn.usembassy.gov/mn/pr-050120-mn/
https://mn.usembassy.gov/mn/pr-042320-mca-website-mn/
https://mn.usembassy.gov/mn/2020-pr-us-mongolia-cpc-mn/
https://mn.usembassy.gov/mn/lead-2020-in-country-mn/
https://mn.usembassy.gov/mn/press-release-usaid-mar-24-2020-mn/
https://mn.usembassy.gov/mn/event-suspension-of-nonimmigrant-and-immigrant-visa-services-due-to-local-covid-19-related-preventative-measures-and-limited-staffing-mn/
https://mn.usembassy.gov/mn/2020-best-program-pr-mn/
https://mn.usembassy.gov/mn/2020-ncov-info-for-visa-mn/

...and so on.
user_100359850
Related