How to get the redirect link of an anchor tag which is built using JavaScript?

user_8211528 0
Viewed 183

I have a link which redirects to an external website... I want to find out what is the final URL that the link redirects to. I tried:

requests.get("link.which.redirects.and.has.dynamic.js.code.com")

But I could not get the final redirected URL as it is built dynamically... I am not sure exactly what happens but there are some JavaScript code involved in loading of the page and the final result is a redirect to an external page.

So instead I tried Selenium and ChromeDriverManager.

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self):
        self.driver = webdriver.Chrome(ChromeDriverManager().install())

    def parse(self, response):
        link = "link.which.redirects.and.has.dynamic.js.code.com"
        self.driver.get(link)
        time.sleep(1) # without this wait, driver.current_url is not the final redirect
        url = self.driver.current_url

The code above loads the entire page, in order to get the redirect URL, is there a more efficient way to get the redirect URL?

python-requests scrapy python
1 Answers

To find out the final like you can use the following code:

import time
import requests

website = requests.get("link.which.redirects.and.has.dynamic.js.code.com", time.sleep(5))

print(website.url) # To see final URL
print(website.history) # To see from where it was the redirection codes
print(website.is_redirect) # Was it redirected 
print(website.is_permanent_redirect) # Is it permanently redirected
user_143183650
Related