bs4 download files even jQuery clicks

user_1981797 0
Viewed 55

I'm trying to automatize a download of subtitles from a public website. The subtitles are accesible once you click on the download link (Descargar in spanish). Inspecting the code of the website, I can see that the links are jQuery events:

enter image description here

There is a function inside this event that, I guess, deals with the download (I'm not at all familiar with JS):

function(a) {
  if (ajaxflagon()) return !1;
  var r = $(this).attr("rel");
  if (r = r.split(","), 3 == r.length) var e = "/updated/" + r[0] + "/" + r[1] + "/" + r[2];
  else var e = "/original/" + r[0] + "/" + r[1];
  ga("send", "pageview", "/" + e, {
    title: "Descargando " + $(this).attr("title")
  }), $(this).attr("href", e), ajaxflagoff()
}

So far, I have code that finds the proper links:

import urllib.request as urlRequest
from bs4 import BeautifulSoup

# Subtitles for a specific TV show
urlpage = 'https://www.tusubtitulo.com/season/4674/1'
# pretend to be a chrome 47 browser on a windows 10 machine
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko)"
}
req = urlRequest.Request(urlpage, headers=headers)
# open the url
url = urlRequest.urlopen(req)
# get the source code
source_code = url.read()

# parse the html using beautiful soup and store in variable 'soup'
soup = BeautifulSoup(source_code, 'html.parser')

results = []
for lang in soup.findAll("td", class_="language"):
    # only interested in the spanish language
    if "Español (España)" in str(lang):
        for element in lang.parent.findAll("a", class_="bt_descarga"):
            results.append(element)

What it's missing is the download part :( How can I do it?

Thanks in advance.

beautifulsoup python
1 Answers

You can implement that JS event function in Python and create the download URLs.

Finally, using the URLs, you can download the subtitles.

Here's how to get the Spanish subs only:

from shutil import copyfileobj

import requests
from bs4 import BeautifulSoup

base_url = "https://www.tusubtitulo.com"
season = "/season/4674/1"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:95.0) Gecko/20100101 Firefox/95.0",
    "Referer": f"{base_url}{season}",
}


def get_rel_attributes(page: str):
    return [
        a["rel"][0] for a in
        BeautifulSoup(page, "lxml").select(".bt_descarga")[1::3]  # this gets only the Spanish subs
    ]


# This is the JS function translated to Python that's responsible for
# building the subtitle download urls.
def get_download_urls(rel_attributes: list):
    src_urls = []
    for item in rel_attributes:
        elements = item.split(",")
        one, two, three = elements
        if len(elements) == 3:
            src_urls.append(f"{base_url}/updated/{one}/{two}/{three}")
        else:
            src_urls.append(f"{base_url}/original/{one}/{two}")
    return src_urls


def downloader(target_url: str, conn: requests.Session):
    response = conn.get(target_url, headers=headers, stream=True)
    file_name = (
        response.headers["Content-Disposition"]
        .split("=", -1)[-1]
        .replace('"', "")
        .encode('latin-1')
        .decode('utf-8')
    )
    print(f"Fetching {file_name}...")
    with open(file_name, "wb") as output:
        copyfileobj(response.raw, output)


if __name__ == "__main__":
    with requests.Session() as connection:
        source_page = connection.get(f"{base_url}{season}", headers=headers).text

    for url in get_download_urls(get_rel_attributes(source_page)):
        downloader(url, connection)

You should see this output and 10 files in the folder you run the script in:

Fetching Invasion (2021) 1x01 - Last Day (Español (España)).srt...
Fetching Invasion (2021) 1x02 - Crash (Español (España)).srt...
Fetching Invasion (2021) 1x03 - Orion (Español (España)).srt...
Fetching Invasion (2021) 1x04 - The King is Dead (Español (Latinoamérica)).srt...
Fetching Invasion (2021) 1x05 - Going Home (Español (España)).srt...
Fetching Invasion (2021) 1x06 - Home Invasion (Español (España)).srt...
Fetching Invasion (2021) 1x07 - Hope (Español (España)).srt...
Fetching Invasion (2021) 1x08 - Contact (Español (España)).srt...
Fetching Invasion (2021) 1x09 - Full of Stars (Español (España)).srt...
Fetching Invasion (2021) 1x10 - First Day (Español (España)).srt...
user_61067910
Related