Python - How to get the page Wikipedia will redirect me to?

user_9021273 0
Viewed 1630

I want to store a few different Wikipedia links but I don't want to store two different links to the same page twice. For example the following links are different but they point to the same Wikipedia page:

https://en.wikipedia.org/w/index.php?title=(1S)-1-Methyl-2,3,4,9-tetrahydro-1H-pyrido-3,4-b-indole&redirect=no 
https://en.wikipedia.org/w/index.php?title=(1S)-1-methyl-2,3,4,9-tetrahydro-1H-pyrido-3,4-b-indole&redirect=no
__________________________________________________|___________________________________________________________

The only difference is that one uppercase character. Or the following links: 

https://en.wikipedia.org/wiki/(0,1)-matrix 
https://en.wikipedia.org/wiki/(0,1)_matrix 
___________________________________|______

That are only different because one has '-' and the other has '_'(' '). So what I want is storing only one of them or the following links: 

https://en.wikipedia.org/wiki/Tetrahydroharman 
https://en.wikipedia.org/wiki/Logical_matrix

I have already tried the answer to this SO question. But it didn't work for me. (The result is the initial URL for me, not the one wiki redirects me to in the browser) So how can I achieve what I'm looking for!?

python-requests url-redirection python-3.x http python
3 Answers

The answer of Amit Tripathi throws an exception. this is my answer:

res = requests.get(url)
doc = lxml.html.fromstring(res.content)
for t in doc.xpath("//link[contains(@rel, 'canonical')]"):
    new_url = str(t.attrib['href'])

from my experience, there might be a redirection to the same url. so better check (url != new_url) before using new_url.

user_107594460
Related