-Python- I can't get redirected link

user_16505965 0
Viewed 38

I want to fetch random scp info with python but it don't give redirected link. It opens a blank page that writes "Random SCP".

#!/usr/bin/env python

import requests
from bs4 import BeautifulSoup as bs

url = "https://scp-wiki.wikidot.com/random:random-scp"
r = requests.get(url, allow_redirects=True)

print(r.url)
python-requests web-scraping python-3.x
1 Answers

The redirect link url is hiding in the JavaScript portion of the content. You can get to it by loading the content into a soup, using css selectors to zero in on it and extracting the final answer with a bit of string manipulation:

soup = soup=bs(r.content,"html.parser")
target = soup.select_one('iframe[src*="redirect"]')
print(target.attrs['src'].split("#")[1])

Output (in this case):

http://scp-wiki.wikidot.com/scp-3794
user_94480900
Related