How to generate X_train for SVR by using for-loop at every iteration of the loop?

Admin 1
Viewed 226

I have datasets that have more than 2000 rows and 23 columns including the age column. I have generated X_train for SVR. Now, I need to generate X_train by taking the rows 2-1001 of the dataset. Basically I shift my X_train by one step on the dataset. And the other data i.e. y_train, X_test, y_test should be shifted as well. Once it is completed I need to do it for the rows 3-1002. And I need to follow this procedure until the end of the dataset. 

#I need to put the below lines  in a for loop
X_train, y_train = data.loc[:1000], target.loc[:1000]
X_test,  y_test  = data.loc[1001], target.loc[1001] 

X_test = np.array(X_test).reshape(1, -1)
print(X_test.shape)


SupportVectorRefModel = SVR()
SupportVectorRefModel.fit(X_train, y_train)

y_pred = SupportVectorRefModel.predict(X_test)
y_pred

#in a for loop and change the indexes in lines 
X_train, y_train = data.loc[:1000], target.loc[:1000] and
X_test,  y_test  = data.loc[1001], target.loc[1001]

that is at every iteration of the loop. Now I am stuck down in this situation. My way of wringing is correct or not? How may I write the code for this condition?

Code from the beginning:

import pandas as pd
import numpy as np

# Make fake dataset
dataset = pd.DataFrame(data= np.random.rand(2000,22))
dataset['age'] = np.random.randint(2, size=2000)

# Separate the target from the other features
target = dataset['age']
data = dataset.drop('age', axis = 1)

X_train, y_train = data.loc[:1000], target.loc[:1000]

X_test,  y_test  = data.loc[1001], target.loc[1001] 

X_test = np.array(X_test).reshape(1, -1)
print(X_test.shape)

SupportVectorRefModel = SVR()
SupportVectorRefModel.fit(X_train, y_train)

y_pred = SupportVectorRefModel.predict(X_test)
y_pred

Code for current work:

for i in range(1, 1001):

        X_train, y_train = dataset.loc[:1000], target.loc[:1000]
        X_test, y_test = dataset.loc[i], target.loc[i]
        X_test = np.array(X_test).reshape(1, -1)
        print(X_test.shape)

        SupportVectorRefModel = SVR()
        SupportVectorRefModel.fit(X_train, y_train)
        y_pred = SupportVectorRefModel.predict(X_test)

Output: KeyError: 1
scikit-learn machine-learning for-loop python loops
1 Answers

Your use case perfectly matches with TimeSeriesSplit cross validation. In your case, each fold size is 1 datapoint.

By default, the training data size would increase by 1 fold in each iteration. See the example here.

You just have to skip the initial 1000 iterations and then use the remaining. since, TimeSeriesSplit returns a generator, you could use the islice to get the elements from 1000 to 1999.

from sklearn.model_selection import TimeSeriesSplit
from itertools import slice

tscv = TimeSeriesSplit(max_train_size=1000, n_splits=1999)

for train_index, test_index in islice(tscv.split(X),1000, 2000):
    <do your stuff here>

Complete example:

import numpy as np
from itertools import islice
from sklearn.model_selection import TimeSeriesSplit
from sklearn.svm import SVR

# Make fake dataset
dataset = pd.DataFrame(data= np.random.rand(2000,22))
dataset['age'] = np.random.randint(2, size=2000)

# Separate the target from the other features
target = dataset['age']
data = dataset.drop('age', axis = 1)

tscv = TimeSeriesSplit(max_train_size=1000, n_splits=1999)

for train_index, test_index in islice(tscv.split(dataset.index),1000, 2000):
    X_train, y_train = dataset.loc[train_index], target.loc[train_index]
    X_test, y_test = dataset.loc[test_index], target.loc[test_index]
    X_test = np.array(X_test).reshape(1, -1)
    SupportVectorRefModel = SVR()
    SupportVectorRefModel.fit(X_train, y_train)
    y_pred = SupportVectorRefModel.predict(X_test)
    print(test_index)

for some reason, if you want to use for loops, try this

for i in range(1, 1001):
    X_train, y_train = data.iloc[i:1000+i], target.iloc[i:1000+i]
    X_test, y_test = data.iloc[1000+i], target.iloc[1000+i]
user_63476290
Related