How do I scrape data from investing.com?

Viewed 5509

I want to scrape the 5 minutes technical summary of EUR/USD from https://au.investing.com/currencies/eur-usd, but I have no idea how to do it. I tried using the requests module, but it said that I was banned from the site.

1 Answers

The server loads the data from external URL. You can use requests module to load it. For example to print candles/times:

import json
import datetime
import requests 
from bs4 import BeautifulSoup


url = 'https://au.investing.com/common/modules/js_instrument_chart/api/data.php?pair_id=1&pair_id_for_news=1&chart_type=area&pair_interval=300&candle_count=120&events=yes&volume_series=yes&period='
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
    'X-Requested-With': 'XMLHttpRequest',
    'Referer': 'https://au.investing.com/currencies/eur-usd'}
data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print( json.dumps(data, indent=4) )

for candle in data['candles']:
    t = datetime.datetime.fromtimestamp(candle[0] // 1000)
    print('{!s:<20} {:<10} {:<10} {:<10}'.format(t, *candle[1:]))

Prints:

2020-10-21 21:40:00  1.1859     123        1004      
2020-10-21 21:45:00  1.1859     184        1127      
2020-10-21 21:50:00  1.1862     173        1311      
2020-10-21 21:55:00  1.186      174        1484      
2020-10-21 22:00:00  1.1863     291        1658      
2020-10-21 22:05:00  1.1864     226        1949      
2020-10-21 22:10:00  1.1863     197        2175      
2020-10-21 22:15:00  1.1863     112        2372      
2020-10-21 22:20:00  1.1863     149        2484      
2020-10-21 22:25:00  1.1862     93         2633      
2020-10-21 22:30:00  1.1863     111        2726      
2020-10-21 22:35:00  1.1861     121        2837      
2020-10-21 22:40:00  1.1862     151        2958      
2020-10-21 22:45:00  1.1861     34         3109      
2020-10-21 22:50:00  1.1861     236        3143      
2020-10-21 22:55:00  1.186      148        3379      
2020-10-21 23:00:00  1.1861     202        3527      
2020-10-21 23:05:00  1.1859     264        3729      
2020-10-21 23:10:00  1.186      162        3993      
2020-10-21 23:15:00  1.1859     223        4155      

...and so on.
Related