How do I scrape the login token generated by JS in Python?

user_11167728 0
Viewed 434

I have to scrape a website, which requires a login token. The value is being replaced later via JS.

document.getElementById('token').value='aa5fedc5decbba3318deab92ffdfbd55d9a2c09ec81a464351ea449dc726ddd5';

The above code is present in the source code after </html> tag. Like so:

</body>
</html>
<script>
    document.getElementById('token').value='aa5fedc5decbba3318deab92ffdfbd55d9a2c09ec81a464351ea449dc726ddd5';
</script>

I have to copy this value and post it to a URL via http request to get request accepted.

But I could not get this value. The code after </html> tag is invisible when I send a request via python requests library.

My python code is here:

import requests
from bs4 import BeautifulSoup

session_requests = requests.session()

html = session_requests.get("http://lms.uaf.edu.pk/login/index.php")
html = html.text
soup = BeautifulSoup(html, "lxml")
print(soup)

How do I get the login token via python?

python javascript
3 Answers

The reason that your code does not work is, that BeautifulSoup skips any html code after </html> (which basically closes the document), so it only parses everything inside the <html> tag, and as the <script> tag with the JavaScript that replaces the login token comes after the closing </html> tag, it won't be parsed by BeautifulSoup and thus you can't get it. In browsers (I tested it on FireFox) everything after the closing </html> tag is moved into the document body, which is why the login token is correctly displayed when you view the site in the browser. You should put BeautifulSoup away for this purpose and just use plain Python (readable version):

import requests
html = requests.get('http://lms.uaf.edu.pk/login/index.php').text # get the html
token_begin = html.find("document.getElementById('token').value='") + len("document.getElementById('token').value='") # find the start of the login token
token_end = html[token_begin:].find("'") # find the end of the login token
token = html[token_begin:token_end]
user_149099800

The following code successfully gets the login token from your website:

import requests
session_requests = requests.session()
html = session_requests.get("http://lms.uaf.edu.pk/login/index.php")
html = html.text
a = html.split("document.getElementById(\'token\').value=\'")[1]
b = a.split("'")[0]
user_108758700

I would go with a straight regex off the response.text, however, it is worth knowing that you can preserve that content, at least with all html parsers I have come across, by wrapping in an outer set of body tags. Not advocating for this approach though in some languages it is very useful for obtaining data in the HEAD which would otherwise get stripped when fed into body.innerHTML of a DOM document.

import requests
from bs4 import BeautifulSoup as bs

r = requests.get('http://lms.uaf.edu.pk/login/index.php')
soup = bs('<body>' + r.text + '</body>', 'lxml')
print(soup.select('script')[-1].string.split("'")[-2])
user_62412350
Related