How can I optimize this data smoothing python loop?

user_16168284 0
Viewed 203

I am trying to make a data smoothng function on a set of data I am using savitzky golay filter in order to do that, I am collecting an array of data and call the function by Scipy. But since I am looping through a spcific element in a different frame I dont have spatial locality nor time locality. dataobj.body.data[j][0][i] holds (x,y) and I am only collecting the ys.

Here's the following loop :

def smooth_data(dataobj):
    number_of_frames = len(dataobj.body.data)
    for i in range(0, 137):
        arr = []
        for j in range(0, number_of_frames):
            arr.append(dataobj.body.data[j][0][i][1])
        newdata = scipy.signal.savgol_filter(arr, 25, 3)
        for k in range(0, number_of_frames):
            dataobj.body.data[k][0][i][1] = newdata[k]
    return dataobj

I'd like to make it work faster, right now when the number of frames is over 1000 it takes a considerable amount of time, something like 30 seconds. Thanks alot to all of the helpers !

smoothing scipy optimization python loops
2 Answers

What about training a Krige model (of just a polynomial interpolation ) with 50 % of your x and y datas, and then taking the ^y evaluation of the model on your whole set x ? Krige model example of code (using smt module) :

from smt.surrogate_models import KRG
t= KRG(theta0=[1e-2]*ndim,print_prediction = False)
t.set_training_values(xt,yt) #training inputs, outputs
t.train()

# Prediction of the other points
y = t.predict_values(xtest)
user_91771780

If the input data is a multi-dimensional numpy array, then you can pass in a slice of the numpy array to the scipy method, and then insert the resulting array back into the original data object:

def smooth_data(dataobj):
    number_of_frames = len(dataobj[:,0,0,1])
    number_of_records = len(dataobj[0,0,:,1])
    for i in range(0, number_of_records):
        newdata = scipy.signal.savgol_filter(dataobj[:,0,i,1], 3, 1)
        dataobj[:][0][i][1] = newdata
    return dataobj
user_1755460
Related