I'm handling option transaction data.
The size of the data is quite big (about 40,000,000 rows × 13 columns), thus it is difficult to compute statistic value (which is essential for my research paper). While I could get mean or variance (+median, 1Q, 3Q ...) easily with the describe() method of pandas DataFrame, skew() and kurt() methods took too much time. Below is the code I tried to use.
skew = pd.DataFrame(df.skew(skipna = True)).T
kurt = pd.DataFrame(df.kurt(skipna = True)+3).T
decs = preprocessed.describe()
desc.append(skew) #took long time
desc.append(kurt) #took long time
Therefore, desc is the table I want to get. I want to know is there a faster way to get skewness and kurtosis value with such big data. Actually, I'm supposed to do the same work repeatedly, it would be very helpful for me.
Thank you for reading the question and I hope you all stay healthy always.