Correlation drop between points where each point has a timeseries

user_17699617 0
Viewed 83

I'm looking into how to make a fast implementation for calculation of correlation between points (in X,Y,Z) and measure when the correlation drops below a certain threshold given an direction of search. I made a previous related question before Calculating correlation between points where each points has a timeseries but I think I need to reformulate the question to make it more clear.

I have time-history of velocity in points in planes fore example XZ-planes where I want to quantify the resolution used for simulations (CFD simulations). To this I want to calculate the Pearson correlation coefficient between the velocity time-history between points and I want to measure for a given axis (For example along the X-axis) the number of points until the correlation drops below a certain threshold (For example 0.8).

I have all the data now in a DataFrame where the head looks like:

    Velocity      X  Y      Z   Time  
0 -12.125850  2.036  0  1.172  10.42
1 -12.516033  2.036  0  1.164  10.42
2 -11.816067  2.028  0  1.172  10.42
3 -10.722124  2.020  0  1.180  10.42
4 -10.628474  2.012  0  1.188  10.42
...

and I want as output this number of points in certain directions, just as an example

    X    Y      Z  CorrDropNrPointsX       CorrDropNrPointsZ       
0 2.036  0  1.172  5                       12
1 2.036  0  1.164  7                       21
2 2.028  0  1.172  4                       18 
3 2.020  0  1.180  12                      19
4 2.012  0  1.188  5                       22
...

In my current implementation I often end up using loops for example .iterrows() which takes forever since I can easily have 300 000 lines in my DataFrame, so I need to speed this up.

I now use lines to detect for a certain direction the number of points along a axis

df_lines = df[[loop_axis]].drop_duplicates()

but the points in planes are not necessarily exactly aligned (there can be small numerical differences in the order of 1-e4 m which can cause errors with this approach and sometimes the points can end because the plane is cut right through a geometry and if that happens the set of points should be viewed of independent of each-other for example for a threshold distance D_L ~ 1-e2 m.

The previous solution I got was quite powerful with calculation of the correlation between all points using

df["cc"] = df.groupby(["X", "Y", "Z"]).cumcount()
df.set_index(["cc","X", "Y", "Z"], inplace=True)
df.unstack(level=[1,2,3])["Velocity"].corr()

however I realized the points are not necessarily ordered in the correct way when loaded in so this needs to be done and for the finally dataframe I still end up looping through rows or columns to measure when the correlation drops below a certain threshold. Another problem with this approach is that it calculates the correlation between all points double so it will calculate corr(P1,P2) and also corr(P2,P1) and also for all points. I will not need to calculate the correlation between diagonal points but only in the defined directions.

All points have the same number of time points so they are sampled with same frequency and same start and end time.

pearson-correlation pandas correlation dataframe python
1 Answers

I think I somewhat understand what you want.

corr_df = df.unstack(level=[1,2,3])["Velocity"].corr()

The correlation matrix can be sorted.

corr_df.sort_index(level=0, inplace=True)

0 for X, 1 for Y , 2 for Z.

You said you wanted the number of points where correlation is greater than .80.

corr_df_x = corr_df.sort_index(level=0)
corr_df_y = corr_df.sort_index(level=1)
corr_df_z = corr_df.sort_index(level=2)

Than you create a series that count the number of consecutive > 0.8. This is a comparison with the first point in columns:

x_df = (df.iloc[:,0].sort_index(level=0)  
       .groupby(df.iloc[:,0].gt(0.8).sort_index(level=0)
       .ne(df.iloc[:,0].gt(0.8).sort_index(level=0).shift())
       .cumsum())
       .transform(lambda x: np.arange(1,len(x)+1)[::-1]))
x_df = x_df.where(df.sort_index(level=0).iloc[:,0].gt(0.8),0)

So x_df will be ordered by X and for each point it will show the number of points larger than 0.8 from that point in the direction of x, the current point is included. Than you would need to merge I think, but I am not sure 100% what you want.

user_8477730
Related