How to use Beautifulsoup to get the latest version of a source package

user_1375531 0
Viewed 68

I have a Bash script which downloads a source file and installs the package. I would like to get the latest version of a package from the website instead of manually updating the script. Beautifulsoup would seem ideal for this. As an example, take the website for Gnome packages -

https://download.gnome.org/sources/

Say I want the latest version of gnome-common. So far I have come up with this:

#!/usr/bin/env python3

from bs4 import BeautifulSoup
import requests
import re

URL="https://download.gnome.org/sources/gnome-common"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
links=soup.find_all("a", href=re.compile("[0-9]"))
print(links)

This gets all the package versions from the site but a whole lot more, so I'm not sure how to proceed. What I want is the list of just the version numbers, then I want to determine the latest version (highest number), then add this to the URL , go to that site and get the latest source tarfile.

So for gnome-common, the latest version is 3.18 and from https://download.gnome.org/sources/gnome-common/3.18/ the latest package is 3.18.0, which is what I will download in the Bash script.

beautifulsoup python
2 Answers

According to the webpage, the latest version is placed at the last position of the table. Therefore you can get it via the index -1. Then you can get the URL via .get_text() method and append it to the current URL. Then, you have to get the link of the tar file. You can use the find_all() function to get all the links in the page. Then you can use a loop to find the file which ends with the extension .tar.xz.

Then use urllib.request to download the file:

#!/usr/bin/env python3

from bs4 import BeautifulSoup
import requests
import re
import urllib.request

tar_file_link = ""
URL="https://download.gnome.org/sources/gnome-desktop/"
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
links=soup.find_all("a", href=re.compile("[0-9]"))
latest_link = links[-1].get_text()
URL+=latest_link
page = requests.get(URL)
soup = BeautifulSoup(page.content, "html.parser")
rows = soup.find_all("a", text=re.compile(".tar.xz"))
tar_file_link = rows[-1].get_text()
if tar_file_link != "":
    URL+=tar_file_link
    print(URL)
    urllib.request.urlretrieve(URL, tar_file_link)
user_69523590

In order to find the last version given that they are already sorted, you can just get the last element of your list of links and append it to the base url since it's a relative url, not absolute.

With links[-1] you can get the last element of your links list which is <a href="3.18/">3.18/</a>.

Then you can append that to your base url to get the url you need:

latest_url = URL + links[-1].text

This will give you the wanted url of the latest version.

Note that you need to have a slash character in between, so I suggest to add a "/" character in your "URL" variable to account for that.

user_96166400
Related