I have been using this code to get stock price data from a selected date range from investing.com for a while now.
But it stopped working all of a sudden and I have no idea why. I looked at the response and it looks different now.
Here is the code I have been using,
import requests
from bs4 import BeautifulSoup
url = "https://www.investing.com/instruments/HistoricalDataAjax"
urlheader = {
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36",
"X-Requested-With": "XMLHttpRequest"
}
localpayload = {'curr_id': '6408',
'st_date': '01/01/2010',
'end_date': '01/01/2020',
'interval_sec': 'Daily',
'action': 'historical_data'}
req = requests.post(url, data=localpayload, headers=urlheader)
soup = BeautifulSoup(req.content, "lxml")
table = soup.find('table', id="curr_table")
This code is to get the Apple Inc. historical data in a table format from here https://www.investing.com/equities/apple-computer-inc-historical-data
I looked at the response from the table and it is None now. So I looked at the soup variable and it looks like this,
<!DOCTYPE html>
<html lang="en-US">
<head>
<title>Just a moment...</title>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
<meta content="IE=Edge" http-equiv="X-UA-Compatible"/>
<meta content="noindex,nofollow" name="robots"/>
<meta content="width=device-width,initial-scale=1" name="viewport"/>
<link href="/cdn-cgi/styles/challenges.css" rel="stylesheet"/>
</head>
<body class="no-js">
<div class="main-wrapper" role="main">
<div class="main-content">
<h1 class="zone-name-title h1">
<img class="heading-favicon" onerror="this.onerror=null;this.parentNode.removeChild(this)" src="/favicon.ico"/>
www.investing.com
</h1>
<h2 class="h2" id="challenge-running">
Checking if the site connection is secure
</h2>
<noscript>
<div id="challenge-error-title">
<div class="h2">
<span class="icon-wrapper">
<div class="heading-icon warning-icon"></div>
</span>
<span id="challenge-error-text">
Enable JavaScript and cookies to continue
</span>
</div>
</div>
</noscript>
<div id="trk_jschal_js" style="display:none;background-image:url('/cdn-cgi/images/trace/managed/nojs/transparent.gif?ray=74d52e315dcd9e22')"></div>
<div class="core-msg spacer" id="challenge-body-text">
www.investing.com needs to review the security of your connection before proceeding.
</div>
<form action="/instruments/HistoricalDataAjax?__cf_chl_f_tk=x_Tq_z13VO2n2YbBpXAQo3VRu2i12yVA.UmxowzgWl8-1663620487-0-gaNycGzNCJE" enctype="application/x-www-form-urlencoded" id="challenge-form" method="POST">
<input name="md" type="hidden" value="f_jOOpoP1gTSxrhsajcvhbmLP1VUiV33bCrsqzCl_eo-1663620487-0-AQGbrPR3dACE-Krs7pU5tXJLLadE9v-DjepvMGjcN-n7Lgdxwq2G-qYpiT3ygWuuurjFwtV5NF5qnn1j0WedpU16awCDYeD99lS7T8W4B9rVi4H1vgbke4y3JlhxqfzqlORriLRdWEo7PfZXai10Aq-j-uWh0nbakKXCPNg4t3fArmnRqycxDR3vqus9KptxD4ZZEHQQusSZtkozp3NYK-0q5C9DRgttjpTs9UnxDiiMuJ-48wcYAt_PmPYegMtjwFfJeg_bmapG4DHODZc3obsTujs4OJMrfc8g3n4gOD-qek69ORe7jE5GC0_VgTS-EvI3rVyzME2ajs19VB0hb9dNuWxM4a-BFr-2cmr9E9YF8WmDF46U_WiivPXK6YC9j9GdBQr8tllOxWEHFt4-rAWcc6ar-bBeUHMM59c71qrcsvA6X3kYJgvk8qTHVFV_YHfj9uU-lR-FiLPd0bbAc7760PHjJ3votfM_hTcYpXJJnVhTz_x3S8m9zjypsvrkXMJpChL5V9EBczCjiSeFwAsYjaCMSelf9ptLZqbbuvb-WeIlqFvuyJ_ELeVlH5dc3wVBpWZolsZqZaIG_aqUA0GPIxZmBZfHYiqwDoIUsBIIiQs98I3hBchWKaM52APBU0b1BRQPykmg4pEr65XhdsZjXoyqdGt0bAJE_Uzx1QnuxVjzUUl8yNbNxVsVDzcrPGAnpr71ls6AD1wn9aHC5YKVag03Rs0NCPRuZ57p4TjW"/>
<input name="r" type="hidden" value="IVrHAaAfQsbgdE_oxUX26rIS0q04IcqW7zUGHsv1X.k-1663620487-0-AaPUvc1mBNw3Er9c7vNcOLkR8nuLorz+EVUoBpQ17t0/cMyyciZPY0aOLnlGmP5r9cfTTe1Lks241GOjS50RVP84qfwQSKUw3ycJNJ42wMiShysFCLOAw1UVhaS+vu0S5bUxNjU8uPL3/REV/q767SsMpZ1HXQWhdRFSfA5tw5ge7rJWkIB387tOOV0r5qgxGW/OjDMg0+Y0jD+mXGpAmMn2R+7aB9KzdTrcrWjasqAdUAN/swc80Cx1LgdAcwZvQIdj9wUb3pjsWD/UrpPNo8jBBEJVt5lGG4my3yxFFoMCyhzFLhz5alU5ltufq3/vW7R1KGjYf3FMY56b1vjrHFBTZeoxOY8d0ic6qw6UtmsAvwX69diQkbvUfw4RSp0w+YN09LaGBaGoHGOtpZz9PmOI2N9DLu9ISMjJLuDDmoTNSTCeMgNZXxHnDkG6xsN9VIqUrA/8h4FPahf+XDnKYev1qL1JDa5QY7y2he0zr5HdrzpdoUeBX3KQVJ4Qv7BBX9U9Dt+47XtnA0vNfm6UHgXGM0f15U3z92yA2kFao9kFOeCNz0x/EY0CuMB3cDlOrIXpreMOGxstt/m6HAJaXBqsD2i4RiBpVT9S7gjOhsiLP2nge/mGPUDpnHFI9LT38Z7QPMp5IxsE3jqIYqlmv5tckvCljbI/CaPWBvY6RiF2D2YiGuWBxQd5wKU6CxwUtxdt3d3Rw84G25BZYNGLOODMZY0EYUYdORymA3WRkOhrJ8XRtfkLfIXdkmgVzWMyIwhpyTJmXGPTH0z+ZhLrq7DKrmZqPlzhtGZHkNf2alY6INKN2iW86NPOnkwFCXHb9KhUR3lfwhXIqluFXtqXyVwzrVf/dzzut4yDg/BHXjwrrLjzkl/cTh01wTE9NqwlqIjR3LiMLXsTXJX/DGjFHimTWj221bicGELwTv8fk5Tv8UlbzOI4pdRARBNxMSmz8AmOX9THImut2aeCkmKsHHuR+AVwTypzSzsOL3pRFA07MVeKdrT3oIyLELlMwit+OE4VKXHKymFT/+Wr3ASZ0n8QlUMauJtrGABsyRcmhh4O0r2hHF4FjUc7/l6YEbh1+lKBKXUQ8aOLRTLi0SJb/VbQtHLH5SRX9EWphYxV31tlt0ha3v8C9H8BuKIRtjXLakZbYUeuw2JFyVjKIlkTF2VtA8lZHl8/miI/IhlBCiWAdH/3tIjooY2mBHeioadwveXIISRbeppmE7bR2dsec+pVzg3IG6fekLFmFUeQu6+SDiKTThf5FYMdSYWF+FKGiVdccIL0r518cUUG/S6QFL8RTH9NjXHkIjJa7dCIGKAfpafptBUT9Cwy/u+hIXgQs1OzMKMJVA64/OZzx4mGShk2+ZmqRvs1tKOdjgd/PZWmsi9hi6sUpt6EDMDtZzAIZHu8rX84amzuOwArr0jRu95Lr0HUaJX+1+BLI3B7UJfdC0ITECtdnWDltKLvqUeeqZHoWqFBZui2E9TbyiOwUQZIRsWr8pB99dyxQgUYZwNiC31Bsaztqxvca554BoqRjbNvWSuNBfb9S59EaFRSAeiuW1WGIOtNZ+eEYvzRywdEE7f/4YJa5W1vjC6zHEFBoNik1tQSgwonnV2xytl/yjplof4mAkQDHfkAk2x8is4rZc+t9GKyJx/a3zNf+ikEvfXa2jnLKh/SykqqDXQ4KkwwAis0Xyb/U+dIRTISakCTuCdmC1NH+9wOEa6BFcCxTxFvEkxv+4A+Lun5JdrJVCMmn1WbBK0+rRxjARDgPUGKV0Um/lBhmE50jG1BrghFVDFe29+2U2GdmOiEUyMb9vusQeoa0BqHnOHVl7qscbNViIfTcMU+QhgqF4oyOoQjEsphsgqlLxDNEGL/+9OIq1a5ap2ucvYpuR/xWGUyGXj+PKhHH0XzrHkrUc57dgk/o650Qrge2BOEfqteyY5mO/yPG/lJUNuicwHYIZfbp1owBzQLYa7C0jlRzZMHSKKoJHUx5fTWx3JE06ucjID9N9Xv9zoU/g7r/I/VZbwNAjD/kr7vGtw6kUvYSpOebgoQXPJ80C5IipAcKb8/PaHw8PI7hIgOGu7frHPL1x0+uDAaKbZ5nvJTPVz3raxQyaTholOwdUSN5ms4PFktsTrYocw="/>
</form>
</div>
</div>
<script>
(function(){
window._cf_chl_opt={
cvId: '2',
cType: 'managed',
cNounce: '75795',
cRay: '74d52e315dcd9e22',
cHash: '77b14089d6597c9',
cUPMDTk: "\/instruments\/HistoricalDataAjax?__cf_chl_tk=x_Tq_z13VO2n2YbBpXAQo3VRu2i12yVA.UmxowzgWl8-1663620487-0-gaNycGzNCJE",
cFPWv: 'b',
cTTimeMs: '1000',
cTplV: 4,
cTplB: 'cf',
cRq: {
ru: 'aHR0cHM6Ly93d3cuaW52ZXN0aW5nLmNvbS9pbnN0cnVtZW50cy9IaXN0b3JpY2FsRGF0YUFqYXg=',
ra: 'TW96aWxsYS81LjAgKFgxMTsgTGludXggeDg2XzY0KSBBcHBsZVdlYktpdC81MzcuMzYgKEtIVE1MLCBsaWtlIEdlY2tvKSBDaHJvbWUvNTAuMC4yNjYxLjc1IFNhZmFyaS81MzcuMzY=',
rm: 'UE9TVA==',
d: '0XwK2GtqKOf6CLWe1C36v4r6rjauuotzBWMLiuOw4l1XN4zCIMyyTb8EdFjn/+ngV8vICpj3WZawaoy+vxbcNFr7a4reoQoht0GDVFg04P/t2UZG54iIUnO8c7ZQjW4HGaxuDtpozUf+xNXozNGuk388Uf/7fdas2bYuujzFrPqLDBcd/zdzS6/dopmNb8/J52dJEkCi8UKhrGVNWmBYHCXC+o27wJ8gh3mP6LB80zTfEepO7Q4bVi4OmIzsRnu5+IxLYWP17nfahZaSztmImL7zuQpOF494H+yi8vyIGDnt3OM7tXslnm/nj5FbwGffQEpMhv2n30UUve8VDF5v6o7L7DBQNYxR4As8cm0okGMnOWR4Iw7pi0+gsF0PJm8IJcvN0NNilXEKnvN82Cm5/823Jair2gW3HMH2b9pa0zv3mWWKa/wejGB1ZgwrA5vJjoXNhi9ViI+eSaxSCGikPs0zb3p7cJK0TpBvKilgZA2wp3EICl/o7pHtzRa2sUDwc6xT9/GDpxWNv9dncM0NLfJVn5X6BcBDSSohSdiXpE7cq4W5Aw56S0c7wtFZvF9XKA7oswGH1J0qd+KQYgX3yngS41Cc3HARSVZl+8OFOypRQzZ7snPQuIQEZJrtx4F11Y7B5y/+g81tFxlSeoUQQcfUyqd7aaF1MEVktWFzb5g=',
t: 'MTY2MzYyMDQ4Ny45MDEwMDA=',
m: 'svId0Tg1UwgusUw+y8yZ6nHVWzIWUczfb7Ui/CXrQCk=',
i1: 'SjlhiBIS+XNVHrAXU/bR8w==',
i2: '87+Z/kOiPCViwl5MRzUBog==',
zh: 'JJQg2KI/+bPgJbLHlLjmrs/mnno8aAGH5k3tm8QDk4c=',
uh: 'ldGBjaEs/Pvt2pvSQ4Vs49DELVSD7b47pQBg1FbRG/4=',
hh: 'HDfVZAmGSp7z9PGIgvFE1PBBQQAUjknIBHsaQqNrk98=',
}
};
var trkjs = document.createElement('img');
trkjs.setAttribute('src', '/cdn-cgi/images/trace/managed/js/transparent.gif?ray=74d52e315dcd9e22');
trkjs.setAttribute('style', 'display: none');
document.body.appendChild(trkjs);
var cpo = document.createElement('script');
cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/managed/v1?ray=74d52e315dcd9e22';
window._cf_chl_opt.cOgUHash = location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;
window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, -window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;
if (window.history && window.history.replaceState) {
var ogU = location.pathname + window._cf_chl_opt.cOgUQuery + window._cf_chl_opt.cOgUHash;
history.replaceState(null, null, "\/instruments\/HistoricalDataAjax?__cf_chl_rt_tk=x_Tq_z13VO2n2YbBpXAQo3VRu2i12yVA.UmxowzgWl8-1663620487-0-gaNycGzNCJE" + window._cf_chl_opt.cOgUHash);
cpo.onload = function() {
history.replaceState(null, null, ogU);
};
}
document.getElementsByTagName('head')[0].appendChild(cpo);
}());
</script>
<div class="footer" role="contentinfo">
<div class="footer-inner">
<div class="clearfix diagnostic-wrapper">
<div class="ray-id">Ray ID: <code>74d52e315dcd9e22</code></div>
</div>
<div class="text-center">Performance & security by <a href="https://www.cloudflare.com?utm_source=challenge&utm_campaign=m" rel="noopener noreferrer" target="_blank">Cloudflare</a></div>
</div>
</div>
</body>
</html>
None
I looked at the network traffic at https://www.investing.com/equities/apple-computer-inc-historical-data and I saw the following website, https://api.investing.com/api/financialdata/historical/6408?start-date=2022-08-19&end-date=2022-09-20&time-frame=Daily&add-missing-rows=false.
If I visit it I get an error {"@errors":["Key: 'Request.Domain' Error:Field validation for 'Domain' failed on the 'required' tag"]}
Any ideas on what went wrong and how I can get around it? I want to extract the table with the historical ohlc data.