python extract titles with full urls with beautifulsoup

user_15700957 0
Viewed 40

I am a beginner Python programmer. For practice, I am trying to get a list of article titles and their URLs from a webpage. So far, I have come up with up with this code:

import requests
from bs4 import BeautifulSoup as BS

with requests.session() as r:
    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0'}
    r = requests.get('https://0xdf.gitlab.io', verify=False, headers=headers)
    response = r.text
    soup = BS(response, 'html.parser')
    tags = soup.find_all('a')

    for tag in tags:
        links = tag.get('href')
        if links[0] == '/':
            appended_link = 'https://0xdf.gitlab.io' + links
            print(appended_link)
        elif links[0] == '#':
            pass
        else:
            print(links)

However, it doesn't extract what I am interested in. I want the title of the articles alongside its full URL.

Thanks

python-requests python
1 Answers

You can use this example how to extract titles from this page + URL:

import requests
from bs4 import BeautifulSoup as BS

url = "https://0xdf.gitlab.io/"
soup = BeautifulSoup(requests.get(url).content, "html.parser")

for link in soup.select(".post-link"):
    print(
        "{:<40} {}".format(
            link.get_text(strip=True), "https://0xdf.gitlab.io" + link["href"]
        )
    )

Prints:

HTB: Toolbox                             https://0xdf.gitlab.io/2021/04/27/htb-toolbox.html
HTB: Bucket                              https://0xdf.gitlab.io/2021/04/24/htb-bucket.html
HTB: Laboratory                          https://0xdf.gitlab.io/2021/04/17/htb-laboratory.html
HTB: APT                                 https://0xdf.gitlab.io/2021/04/10/htb-apt.html
HTB: Time                                https://0xdf.gitlab.io/2021/04/03/htb-time.html
HTB: Luanne                              https://0xdf.gitlab.io/2021/03/27/htb-luanne.html
HTB: CrossFit                            https://0xdf.gitlab.io/2021/03/20/htb-crossfit.html
HTB: Optimum                             https://0xdf.gitlab.io/2021/03/17/htb-optimum.html
Reel2: Root Shell                        https://0xdf.gitlab.io/2021/03/15/reel2-root-shell.html
HTB: Reel2                               https://0xdf.gitlab.io/2021/03/13/htb-reel2.html
HTB: Sense                               https://0xdf.gitlab.io/2021/03/11/htb-sense.html
HTB: Passage                             https://0xdf.gitlab.io/2021/03/06/htb-passage.html
HTB: Sneaky                              https://0xdf.gitlab.io/2021/03/02/htb-sneaky.html
HTB: Academy                             https://0xdf.gitlab.io/2021/02/27/htb-academy.html
HTB: Beep                                https://0xdf.gitlab.io/2021/02/23/htb-beep.html
HTB: Feline                              https://0xdf.gitlab.io/2021/02/20/htb-feline.html
HTB: Charon                              https://0xdf.gitlab.io/2021/02/16/htb-charon.html
HTB: Jewel                               https://0xdf.gitlab.io/2021/02/13/htb-jewel.html
HTB: Apocalyst                           https://0xdf.gitlab.io/2021/02/09/htb-apocalyst.html
HTB: Doctor                              https://0xdf.gitlab.io/2021/02/06/htb-doctor.html
HTB: Europa                              https://0xdf.gitlab.io/2021/02/02/htb-europa.html
HTB: Worker                              https://0xdf.gitlab.io/2021/01/30/htb-worker.html
HTB: Compromised                         https://0xdf.gitlab.io/2021/01/23/htb-compromised.html
HTB: RopeTwo                             https://0xdf.gitlab.io/2021/01/16/htb-ropetwo.html

...and so on.
user_100359850
Related