How can I send a calendar choice in a URL as a header using requests?

user_10418924 0
Viewed 62

I am trying to use BeautifulSoup in this URL: https://es.investing.com/dividends-calendar/

My problem is that I want to set a specific dates to show. I saw in chrome network options what header is sending. But I had no luck getting the correct page, I am asking the URL for data from 01/01/2019 to 12/31/2020, but I always getting 2021 data.

This is what I tryed for now:

from bs4 import BeautifulSoup
import requests
from urllib.parse import parse_qsl

session = requests.Session()
session.headers.update({'User-Agent':'Mozilla/5.0'})

qs = 'country%5B%5D=26&dateFrom=2019-01-01&dateTo=2020-12-31&currentTab=custom&limit_from=0'
payload = dict(parse_qsl(qs))

html_text = session.post('https://es.investing.com/dividends-calendar/', data=payload).text
soup = BeautifulSoup(html_text,'lxml')
job = soup.find_all('table')

This qs is (I guess) the data from is sending to show the correct calendar range, this qs is what I got from Chrome Network Options.

enter image description here

Any idea what I am doing wrong??

Tanks a lot!

python-requests web-scraping beautifulsoup python
1 Answers

You were missing a few other required headers and also the URL for the data was different than the main URL.

from bs4 import BeautifulSoup
import requests

session = requests.Session()

headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0",
    "Accept" : "*/*",
    "Accept-Language" : "en-GB,en;q=0.7,de;q=0.3",
    "Referer": "https://es.investing.com/dividends-calendar/",
    "Content-Type" : "application/x-www-form-urlencoded",
    "X-Requested-With" : "XMLHttpRequest",
    "Origin" : "https://es.investing.com",
}

payload = {
    'country[]': [26, 5, 22, 4], 
    'dateFrom': '2020-12-01', 
    'dateTo': '2020-12-24', 
    'currentTab': 'custom', 
    'limit_from': 0
}

r = session.post('https://es.investing.com/dividends-calendar/Service/getCalendarFilteredData', data=payload, headers=headers)
j = r.json()

soup = BeautifulSoup(j['data'], 'lxml')

for row in soup.find_all('tr')[1:]:
    values = [td.text for td in row.find_all('td')]
    print(values[1:])

Giving you output starting:

['Annaly Capital Management Pd Pref\xa0(NLY_pd)', '01.12.2020', '0,46875', '', '31.12.2020', '7,50%']
['Kimberly-Clark de Mexico\xa0(KCDMY)', '01.12.2020', '0,461541', '', '--', '5,83%']
['Thales\xa0(TCFP)', '01.12.2020', '0,4', '', '03.12.2020', '0,53%']
['McCormick&Co\xa0(MKC)', '01.12.2020', '0,34', '', '30.11.2020', '1,45%']
['McCormick & Comp\xa0(MKCv)', '01.12.2020', '0,34', '', '30.11.2020', '1,48%']
['Goldman Sachs\xa0(GS)', '01.12.2020', '1,25', '', '30.12.2020', '1,66%']
['Schlumberger\xa0(SLB)', '01.12.2020', '0,125', '', '14.01.2021', '1,97%']
['Avery Dennison\xa0(AVY)', '01.12.2020', '0,62', '', '16.12.2020', '1,55%']
['Ardagh Group\xa0(ARD)', '01.12.2020', '0,15', '', '16.12.2020', '3,45%']
user_49857330
Related