I want to scrape the 5 minutes technical summary of EUR/USD from https://au.investing.com/currencies/eur-usd, but I have no idea how to do it. I tried using the requests module, but it said that I was banned from the site.
I want to scrape the 5 minutes technical summary of EUR/USD from https://au.investing.com/currencies/eur-usd, but I have no idea how to do it. I tried using the requests module, but it said that I was banned from the site.
The server loads the data from external URL. You can use requests module to load it. For example to print candles/times:
import json
import datetime
import requests
from bs4 import BeautifulSoup
url = 'https://au.investing.com/common/modules/js_instrument_chart/api/data.php?pair_id=1&pair_id_for_news=1&chart_type=area&pair_interval=300&candle_count=120&events=yes&volume_series=yes&period='
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
'X-Requested-With': 'XMLHttpRequest',
'Referer': 'https://au.investing.com/currencies/eur-usd'}
data = requests.get(url, headers=headers).json()
# uncomment this to print all data:
# print( json.dumps(data, indent=4) )
for candle in data['candles']:
t = datetime.datetime.fromtimestamp(candle[0] // 1000)
print('{!s:<20} {:<10} {:<10} {:<10}'.format(t, *candle[1:]))
Prints:
2020-10-21 21:40:00 1.1859 123 1004
2020-10-21 21:45:00 1.1859 184 1127
2020-10-21 21:50:00 1.1862 173 1311
2020-10-21 21:55:00 1.186 174 1484
2020-10-21 22:00:00 1.1863 291 1658
2020-10-21 22:05:00 1.1864 226 1949
2020-10-21 22:10:00 1.1863 197 2175
2020-10-21 22:15:00 1.1863 112 2372
2020-10-21 22:20:00 1.1863 149 2484
2020-10-21 22:25:00 1.1862 93 2633
2020-10-21 22:30:00 1.1863 111 2726
2020-10-21 22:35:00 1.1861 121 2837
2020-10-21 22:40:00 1.1862 151 2958
2020-10-21 22:45:00 1.1861 34 3109
2020-10-21 22:50:00 1.1861 236 3143
2020-10-21 22:55:00 1.186 148 3379
2020-10-21 23:00:00 1.1861 202 3527
2020-10-21 23:05:00 1.1859 264 3729
2020-10-21 23:10:00 1.186 162 3993
2020-10-21 23:15:00 1.1859 223 4155
...and so on.