Making a requests.post to access a subpage with BeautifulSoup?

user_8157304 0
Viewed 205

Using BeautifulSoup on Python, I'm trying to scrape a subpage of this page

https://www.mmorpg-stat.eu/0_fiche_alliance.php?pays=5&ftr=500208.all&univers=_146

More precisely, the subpage titled OTHER INFORMATION whose button has this code

onclick="fcache12('faCacher');fcache13('ffond_gris');
    document.form1_2date.statview.value='2';
    document.forms['form1_2date'].submit();return false;"

which calls the methods fcache12() and fcache13() and then it finds the form form1_2date and submits it using the value value='2' (values 0 and 1 are for the previous two buttons)

<form name="form1_2date" method="post">

The problem is that by clicking on the button the url doesn't change so the page cannot be reached with requests.get(), but requests.post() should be used instead.

import requests, urllib.request
from bs4 import BeautifulSoup

url = 'https://www.mmorpg-stat.eu/0_fiche_alliance.php?pays=5&ftr=500208.all&univers=_146'
input = {} # what to put here?
response = requests.post(url, data = input)

I read that to simulate a click on the button I have to determine the form values that should be passed in, and that these are determined by all of the <input> tags in the form.

There are 12 <input>s inside form1_2date

1 <input name="keyf" id="keyf" type="hidden">
2 <input type="checkbox" name="zoomgraph_box" id="zoomgraph_box">
3 <input type="checkbox" name="zoomgraph_box" id="zoomgraph_box">
4 <input name="Submit" type="button" onclick="fcache12('faCacher');fcache13('ffond_gris');document.getElementById('menu_live_a').style.display='none';
document.form1_info3.zoomgraph.value=document.getElementsByName('zoomgraph_box')[0].checked;document.form1_info3.choixflash.value=document.getElementsByName('zoomgraph_box')[1].checked;
document.forms['form1_info3'].submit();return false;" style="font:9pt Arial, Helvetica, sans-serif;border: thin solid #666666; cursor:pointer; background:#EA6C11; color: #ffffff;-moz-border-radius: 10px;-webkit-border-radius: 10px;border-radius: 6px 6px 6px 6px; " value="Apply">
5 <input name="Submit2" type="button" onclick="document.getElementById('menu_live_a').style.display='none';" style="font:9pt Arial, Helvetica, sans-serif;border: thin solid #666666; cursor:pointer;background:#EA6C11; color: #ffffff;-moz-border-radius: 10px;-webkit-border-radius: 10px;border-radius: 6px 6px 6px 6px; " value="Cancel">
6 <input name="date_1" type="text" class="case_ss1 hasDatepicker" id="datepickera" value="13-10-2019" size="10" maxlength="10">
7 <input name="date_2" type="text" class="case_ss1 hasDatepicker" id="datepicker" value="19-10-2019" size="10" maxlength="10">
8 <input name="plot_1" type="hidden" id="plot_1">
9 <input name="Submit" type="button" onclick="fcache12('faCacher');fcache13('ffond_gris');document.getElementById('cache_date_j_div').style.display='none';document.form1_2date.plot_1.value='6';document.form1_2date.keyf.value='';document.forms['form1_2date'].submit();return false;" style="font:9pt Arial, Helvetica, sans-serif;border: thin solid #666666; cursor:pointer; background:#EA6C11; color: #ffffff;-moz-border-radius: 10px;-webkit-border-radius: 10px;border-radius: 6px 6px 6px 6px; " value="Apply">
10 <input name="Submit2" type="button" onclick="document.getElementById('cache_date_j_div').style.display='none';" style="font:9pt Arial, Helvetica, sans-serif;border: thin solid #666666; cursor:pointer; background:#EA6C11; color: #ffffff;-moz-border-radius: 10px;-webkit-border-radius: 10px;border-radius: 6px 6px 6px 6px; " value="Cancel">
11 <input name="statview" type="hidden" id="statview" value="">
12 <input name="statview_bis" type="hidden" id="statview_bis" value="">

I read that the dictionary has to be written in the form {'key1': value1, 'key2': value2, ...} but I don't understand how to compile it.

beautifulsoup python
1 Answers

In your web browser, open the dev tools (you can do so by pressing F12)

Then open the network tab, there you will see every request your browser make.

Now, click on the Other information button on the webpage. So then, in the dev tools you'll see a few requests being made, the one that you need is this one:

Dev tools screenshot

When you select it, it opens the details of the request on the right side panel.

So to make your request work, you have to mimic this; on the screenshot is shown the Params tab, which is probably the most important, but there are also cookies you'll need to get (you can use requests.Session for the PHPSESSID)

This works :

import requests

response = requests.post(
        'https://www.mmorpg-stat.eu/0_fiche_alliance.php?pays=5&ftr=500208.all&univers=_146',
        data = {
            'date_1': '01-10-2019',
            'date_2': '19-10-2019',
            'statview': 2,
        }
    )

print(response.text)
user_33224000
Related