Running Scipy Linregress Across Dataframe Where Each Element is a List

I am newbie to this but I do optimization your original code:

• by purely use python builtin list object (there is no need to use pandas and to be honest I cannot find a better way to solve your problem in pandas than you original code :D)

• by using numpy, which should be (at least they claimed) faster than python builtin list.

You can jump to see the code, its in Jupyter notebook format so you need to install Jupyter first.

Conclusion

Here is the test result: On a (100, 100) matrix containing (30,) length random lists, the total time difference is around 1 second.

Time elapsed to run 1 times on new method is 24.282760 seconds.
Time elapsed to run 1 times on old method is 25.954801 seconds.

Refer to test_perf in sample code for result.

PS: During test only one thread is used, so maybe multi-thread will help to improve performance, but that's out of my ability...

Idea

I think numpy.nditer is suitable for your request, though the result of optimization is not that significant. Here is my idea:

1. Generate the input array

   I have altered you first part of script, I think using list comprehension along is enough to build a matrix of random lists. Refer to get_matrix_from_builtin. Please note I have stored the random lists in another 1-element tuple to keep the shape as ndarray generate from numpy.

   As a compare, you can also construct such matrix with numpy. Refer to get_matrix_from_numpy. Because ndarray try to boardcast list-like object (and I don't know how to stop it), I have to wrap it into a tuple to avoid auto boardcast from numpy.array constructor. If anyone have a better solution please note it, thanks :)

2. Calculate the result

   I altered you original code using pandas.DataFrame to access element by row/col index, but it is not that way. Pandas provides some iteration tool for DataFrame: pipe, apply, agg, and appymap, search API for more info, but it seems not suitable for your request here, as you want to obtain the current index of row and col during iteration.

   I searched and found numpy.nditer can provide that needs: it return a iterator of ndarray, which have an attribution multi_index that provide the row/col pair of current element. see iterating-over-arrays

Explain on solve.ipynb

I use Jupyter Notebook to test this, you might need got one, here is the instruction of install.

I have altered your original code, which remove the request of pandas and purely used builtin list. Refer to old_calc_tstat in the sample code.

Also, I used numpy.nditer to calc your tstats matrix, Refer to new_calc_tstat in the sample code.

Then, I tested if the result of both methods are equal, I used same input array to ensure random won't affect the test. Refer to test_equal for result.

Finally, do the time performance. I am not patient so I only run it for one time, you may add the repeats count of test in the test_perf function.

The code

# To add a new cell, type '# %%'
# To add a new markdown cell, type '# %% [markdown]'
# %% [markdown]
# [origin question](https://stackoverflow.com/questions/69228572/running-scipy-linregress-across-dataframe-where-each-element-is-a-list)
#

# %%
import sys
import time
import numpy as np
from scipy.stats import linregress


# %%
def get_matrix_from_builtin():
    # use builtin list to construct matrix of random list
    # note I put random list inside a tuple to keep it same shape
    # as I later use numpy to do the same thing.
    return [
        [(list(np.random.rand(11)),)
         for col in range(6)]
        for row in range(100)
    ]


# %timeit get_matrix_from_builtin()


# %%
def get_matrix_from_numpy(
    gen=np.random.rand,
    shape=(1, 1),
    nest_shape=(1, ),
):

    # custom dtype for random lists
    mydtype = [
        ('randonlist', 'f', nest_shape)
    ]
    a = np.empty(shape, dtype=mydtype)
    # [DOC] moditfying array values
    # https://numpy.org/doc/stable/reference/arrays.nditer.html#modifying-array-values
    # enable per operation flags 'readwrite' to modify element in ndarray
    # enable global flag 'refs_ok' to allow use callable function 'gen' in iteration
    with np.nditer(a, op_flags=['readwrite'], flags=['refs_ok']) as it:
        for x in it:
            # pack list in a 1-d turple to prevent numpy boardcast it
            x[...] = (gen(nest_shape[0]), )

    return a


def test_get_matrix_from_numpy():
    gen = np.random.rand  # generator of random list
    shape = (6, 100)      # shape of matrix to hold random lists
    nest_shape = (11, )   # shape of random lists
    return get_matrix_from_numpy(gen, shape, nest_shape)
# access a random list by a[row][col][0]


# %timeit test_get_matrix_from_numpy()


# %%
def test_get_matrix_from_numpy():
    gen = np.random.rand
    shape = (6, 100)
    nest_shape = (11, )
    return get_matrix_from_numpy(gen, shape, nest_shape)


# %%
def old_calc_tstat(a=None):
    if a is None:
        a = get_matrix_from_builtin()
        a = np.array(a)
    rows, cols = a.shape[:2]
    tstats = np.zeros(shape=(rows, cols))
    for i in range(0, rows):
        for j in range(1, cols):
            lg = linregress(a[i][0][0], a[i][j][0])
            tstats[i, j-1] = lg.slope/lg.stderr

    return tstats


# %%
def new_calc_tstat(a=None):
    # read input metrix of random lists
    if a is None:
        gen = np.random.rand
        shape = (6, 100)
        nest_shape = (11, )
        a = get_matrix_from_numpy(gen, shape, nest_shape)
    # construct ndarray for t-stat result
    tstats = np.empty(a.shape)
    # enable global flags 'multi_index' to retrive index of current element
    # [DOC] Tracking an Index or Multi-Index
    # https://numpy.org/doc/stable/reference/arrays.nditer.html#tracking-an-index-or-multi-index
    it = np.nditer(tstats, op_flags=['readwrite'], flags=['multi_index'])
    # obtain total columns count of tstats's shape
    col = tstats.shape[1]
    for x in it:
        i, j = it.multi_index
        # trick to avoid IndexError: substract len(list) after +1 to index
        j = j + 1 - col
        lg = linregress(
            a[i][0][0],
            a[i][j][0]
        )
        # note: nditer ignore ZeroDivisionError by default, and return np.inf to the element
        # you have to override it manually:
        if lg.stderr == 0:
            x[...] = 0
        else:
            x[...] = lg.slope / lg.stderr
    return tstats


# new_calc_tstat()


# %%
def test_equal():
    """Test if the new method has equal output to old one"""
    # use same input list to avoid affect of rand
    a = test_get_matrix_from_numpy()
    old = old_calc_tstat(a)
    new = new_calc_tstat(a)
    print(
        "Is the shape of old and new same ?\n%s. old: %s, new: %s\n" % (
            old.shape == new.shape, old.shape, new.shape),
    )
    res = (old == new)
    print(
        "Is the result object same?"
    )
    if res.all() == True:
        print("True.")
    else:
        print("False. Difference(new - old) as below:\n")
        print(new - old)

    return old, new


old, new = test_equal()


# %%
# the only diff is the last element
# in old method it is 0
# in new method it is inf
# if you perfer the old method, just add condition in new method to override
# [new[x][99] for x in range(6)]


# %%
# python version: 3.8.8

timer = time.clock if sys.platform[:3] == 'win' else time.time


def total(func, *args, _reps=1, **kwargs):
    start = timer()
    for i in range(_reps):
        ret = func(*args, **kwargs)
    elapsed = timer() - start
    return elapsed


def test_perf():
    """Test of performance"""
    # first, get a larger input array
    gen = np.random.rand
    shape = (1000, 100)
    nest_shape = (30, )
    a = get_matrix_from_numpy(gen, shape, nest_shape)

    # repeat how many time for each test
    reps = 1

    # then, time both old and new calculation method
    old = total(old_calc_tstat, a, _reps=reps)
    new = total(new_calc_tstat, a, _reps=reps)

    msg = "Time elapsed to run %d times on %s is %f seconds."
    print(msg % (reps, 'new method', new))
    print(msg % (reps, 'old method', old))


test_perf()