Getting BTC historical data by Kraken API

user_13520255 0
Viewed 946

I'm trying to getting data from kraken exchange by the krakenex API. But i'm facing several problems, 'cause, I want getting the data in a range time bigger than the alllowed by the API. The API only allows getting a dataframe with 720 rows, so 'cause that I need to do a loop while to getting more data and concat in another dataframe.

I've already read other topics about it, but I'm still not reaching good results.

import krakenex
import time
import krakenex
import pandas as pd
from pykrakenapi import KrakenAPI
from datetime import datetime

k = krakenex.API()

start = '28/01/2021 00:00:00'
start = datetime.strptime(start, "%d/%m/%Y %H:%M:%S")
start = int(time.mktime(start.timetuple()))

stop = '03/02/2021 00:00:00'
stop = datetime.strptime(stop, "%d/%m/%Y %H:%M:%S")
stop = int(time.mktime(stop.timetuple()))


prices = pd.DataFrame()



while start < stop:
    
    time.sleep(5)
    data = k.query_public('OHLC', {'pair':'XXBTZUSD', 'interval':1, 'since':start})
    df = pd.DataFrame( data['result']['XXBTZUSD']) 
    daily_prices = df[0].to_list()
    start = int(daily_prices[0])
    prices = pd.concat([precos , df])
pandas python
1 Answers

For weeks I have been working on a script that does exactly that. In my case I collect all pairs with BTC and ETH but you can use the script with any pair. To do this I used the REST API and defined some functions that automate everything. I download the data with 1 minute timeframe but it can be used for any timeframe.

First I defined a function that downloads the data in full or from a specific date, it's necessary because at the first run it will download all the data and then it will download only the new data. The parameter 'interval' defines the number of minutes of the timeframe while 'since' defines the beginning of the data to download.

def get_ohlc (pair, interval=1, since='last'):
    endpoint = 'https://api.kraken.com/0/public/OHLC'
    payLoad = {
        'pair':     pair,
        'interval': interval,
        'since' :   since
    }
    response = requests.get(endpoint, payLoad)
    data = response.json()
    OHLC = data['result'][pair]
    data = pd.DataFrame.from_records(OHLC, columns=['Time', 'Open', 'High', 'Low', 'Close', 'vwap', 'volume', 'count'])
    data['Time'] = pd.to_datetime(data['Time'], unit='s')
    data.set_index('Time',inplace=True)
    data = data.drop(['vwap', 'volume', 'count'], axis=1)
    data['Open']  = data.Open.astype(float)
    data['High']  = data.High.astype(float)
    data['Low']   = data.Low.astype(float)
    data['Close'] = data.Close.astype(float)
    return data

Then I defined a function to load the .json file that was saved into memory. The function returns the dataframe with the old data and a timestamp that indicates from where to download the new data. I also created a function for calculate the timestamp.

def load_data(pair, path):
    data = pd.read_json(path + pair + '.json' , orient='split')
    tmp = data.tail(1).index                    
    tmp = tmp.strftime('%Y-%m-%d %H:%M:%S')     
    dt = str_to_datetime(tmp[0])                
    ts = dt.timestamp()                         
    return data, ts

def str_to_datetime(datestr):
    Y = int(datestr[0:4])
    M = int(datestr[5:7])
    D = int(datestr[8:10])
    H = int(datestr[11:13])
    m = int(datestr[14:16])

    return datetime.datetime(Y, M, D, H, m, 0, tzinfo=tz.gettz("Etc/GMT"))

Now your main should be something like:

from countdown import countdown
import pandas as pd
import datetime
import os

path = os.getcwd() + '/historical_data/'
pair = 'XBTUSD'

while True:
    if os.path.exists(path + pair + '.json') == False:
        data = get_ohlc(pair, 1) # 1 minute timeframe
        data.to_json(path + pair + '.json', orient='split')
    else:
        data1, ts = load_data(pair, path)
        data2 = get_ohlc(pair, 1, ts)
        data3 = pd.concat([data1, data2])
        data3.drop(data3.tail(1).index,inplace=True) # delete last record because it's not ended
        data3.to_json(path + pair + '.json', orient='split')

    countdown(60) # update every hour

I delete the last record because when you download it it's not ended so we will download at the next update. I haven't tested if it works because I took pieces of code from my program, if it doesn't work let me know and I'll fix it.

user_70353060
Related