Encapsulating Vectorised Functions - For Use With Panda DataFrames

Dems_53341 0
Viewed 186

I've been re-factoring some code and using it to explore how to structure maintainable, flexible, concise code when using Pandas and Numpy. (Usually I only use them briefly, I'm now in a role where I should be aiming to become an ex-spurt.)

One example I came across is a function that can sometimes be called on one column of values, and sometimes called on three columns of values. Vectorised code using Numpy encapsulated it wonderfully. But using it becomes a bit clunky.

How should I "better" write the following function?

def project_unit_space_to_index_space(v, vertices_per_edge):
    return np.rint((v + 1) / 2 * (vertices_per_edge - 1)).astype(int)


input = np.concatenate(([df['x']], [df['y']], [df['z']]), axis=0)

index_space = project_unit_space_to_index_space(input, 42)

magic_space = some_other_transformation_code(index_space, foo, bar)

df['x_'], df['y_'], df['z_'] = magic_space

As written the function can accept one column of data, or many columns of data, and it still works correctly, and speedily.

The return type is the right shape to be passed directly in to another similarly structured function, allowing me to chain functions neatly.

Even assigning the results back to new columns in a dataframe isn't "awful", though it is a little clunky.

But packaging up the inputs as a single np.ndarray is very very clunky indeed.


I haven't found any style guides that cover this. They're all over itterrows and lambda expressions, etc. But I found nothing on best practices for encapsulating such logic.


So, how you you structure the above code?


EDIT: Timings of various options for collating the inputs

%timeit test = project_unit_sphere_to_unit_cube(df[['x','y','z']].unstack().to_numpy())                      
# 1.44 ms ± 57.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit test = project_unit_sphere_to_unit_cube(df[['x','y','z']].to_numpy().T)                              
# 558 µs ± 6.25 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit test = project_unit_sphere_to_unit_cube(df[['x','y','z']].transpose().to_numpy())                    
# 817 µs ± 18.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

%timeit test = project_unit_sphere_to_unit_cube(np.concatenate(([df['x']], [df['y']], [df['z']]), axis=0))   
# 3.46 ms ± 42.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
pandas vector encapsulation numpy python
2 Answers
In [101]: df = pd.DataFrame(np.arange(12).reshape(4,3))                         
In [102]: df                                                                    
Out[102]: 
   0   1   2
0  0   1   2
1  3   4   5
2  6   7   8
3  9  10  11

You are making a (n,m) array from n columns of the dataframe:

In [103]: np.concatenate([[df[0]],[df[1]],[df[2]]],0)                           
Out[103]: 
array([[ 0,  3,  6,  9],
       [ 1,  4,  7, 10],
       [ 2,  5,  8, 11]])

a more compact way to do this is transpose the array of those columns:

In [104]: df.to_numpy().T                                                       
Out[104]: 
array([[ 0,  3,  6,  9],
       [ 1,  4,  7, 10],
       [ 2,  5,  8, 11]])

The dataframe has its own transpose:

In [109]: df.transpose().to_numpy()                                             
Out[109]: 
array([[ 0,  3,  6,  9],
       [ 1,  4,  7, 10],
       [ 2,  5,  8, 11]])

Your calculation works with a dataframe, returning a dataframe with same shape and indices:

In [113]: np.rint((df+1)/2 *(42-1)).astype(int)                                 
Out[113]: 
     0    1    2
0   20   41   62
1   82  102  123
2  144  164  184
3  205  226  246

Some numpy functions convert the inputs to numpy arrays and return an array. Others, by delegating details to pandas methods, can work directly on the dataframe, and return a dataframe.

user_9019250

I don't like accepting my own answers, so I'm not going to change the accepted answer.

@hpaulj helped me explore this problem further by making additional functionality and opportunities clear to me. This helped me more clearly define my competing objectives, and also to be able to begin ascribing priority to them.

  1. Code should be terse / compact and maintainable, not full of boiler plate, including...

    • invoking the function
    • utilising the results
    • the function implementation itself

  2. Function performance should not be compromised

    • being 5% slower but better in every other respect may be acceptable
    • being 100% slower is probably never acceptable

  3. Implementation should be as data-type agnostic as possible

    • one function for scalars and another for vectors is less than ideal

This lead me to my currently preferred implementation / style...

def scale_unit_cube_to_unit_sphere(*values):
    """
    Scales all the inputs (on a row basis for array_line types) such that when
    treated as n-dimensional vectors, their scale is always 1.

    (Divides the vector represented by each row of inputs by that row's
     root-of-sum-of-squares, so as to normalise to a unit magnitude.)

    Examples - Scalar Inputs
    --------

    >>> scale_unit_cube_to_unit_sphere(1, 1, 1)
    [0.5773502691896258, 0.5773502691896258, 0.5773502691896258]

    Examples - Array Like Inputs
    --------

    >>> x = [ 1, 2, 3]
    >>> y = [ 1, 4, 3]
    >>> z = [ 1,-3,-1]
    >>> scale_unit_cube_to_unit_sphere(x, y, z)
    [array([0.57735027, 0.37139068, 0.6882472 ]),
     array([0.57735027, 0.74278135, 0.6882472 ]),
     array([ 0.57735027, -0.55708601, -0.22941573])]

    >>> a = np.array([x, y, z])
    >>> scale_unit_cube_to_unit_sphere(*a)
    [array([0.57735027, 0.37139068, 0.6882472 ]),
     array([0.57735027, 0.74278135, 0.6882472 ]),
     array([ 0.57735027, -0.55708601, -0.22941573])]

    scale_unit_cube_to_unit_sphere(*t)
    >>> t = (x, y, z)
    >>> scale_unit_cube_to_unit_sphere(*t)
    [array([0.57735027, 0.37139068, 0.6882472 ]),
     array([0.57735027, 0.74278135, 0.6882472 ]),
     array([ 0.57735027, -0.55708601, -0.22941573])]

    >>> df = pd.DataFrame(data={'x':x,'y':y,'z':z})
    >>> scale_unit_cube_to_unit_sphere(df['x'], df['y'], df['z'])
    [0    0.577350
     1    0.371391
     2    0.688247
     dtype: float64,
     0    0.577350
     1    0.742781
     2    0.688247
     dtype: float64,
     0    0.577350
     1   -0.557086
     2   -0.229416
     dtype: float64]

    For all array_like inputs, the results can then be utilised in similar
    ways, such as writing them to an existing DataFrame as follows:

    >>> transform = scale_unit_cube_to_unit_sphere(df['x'], df['y'], df['z'])
    >> df['i'], df['j'], df['k'] = transform

    """
    # Scale the position in space to be a unit vector, as on the surface of a sphere
    ################################################################################

    scaler = np.sqrt(sum([np.multiply(v, v) for v in values]))
    return [np.divide(v, scaler) for v in values]

As per the doc string, this works with Scalars, Arrays, Series, etc, whether providing one Scalar, three Scalars, n-scalars, n-Arrays, etc.

(I don't yet have a neat and tidy way of passing in a single DataFrame rather than three distinct DataSeries, but that's a low priority for now.)

They also work in "chains" such as the example below (the functions' implementations not being relevant, just the pattern of chaining input to output)...

cube, ix = generate_index_cube(vertices_per_edge)

df = pd.DataFrame(
         data  = {
             'x': cube[0],
             'y': cube[1],
             'z': cube[2],
         },
         index = ix,
     )

unit = scale_index_to_unit(vertices_per_edge, *cube)

distortion = scale_unit_to_distortion(distortion_factor, *unit)

df['a'], df['b'], df['c'] = distortion

sphere = scale_unit_cube_to_unit_sphere(*distortion)

df['i'], df['j'], df['k'] = sphere

recovered_distortion = scale_unit_sphere_to_unit_cube(*sphere)

df['a_'], df['b_'], df['c_'] = recovered_distortion

recovered_cube = scale_unit_to_index(
                     vertices_per_edge,
                     *scale_distortion_to_unit(
                         distortion_factor,
                         *recovered_distortion,
                     ),
                 )

df['x_'], df['y_'], df['z_'] = recovered_cube

print(len(df[np.logical_not(np.isclose(df['a'], df['a_']))]))  # No Differences
print(len(df[np.logical_not(np.isclose(df['b'], df['b_']))]))  # No Differences
print(len(df[np.logical_not(np.isclose(df['c'], df['c_']))]))  # No Differences

print(len(df[np.logical_not(np.isclose(df['x'], df['x_']))]))  # No Differences
print(len(df[np.logical_not(np.isclose(df['y'], df['y_']))]))  # No Differences
print(len(df[np.logical_not(np.isclose(df['z'], df['z_']))]))  # No Differences

Please do comment or critique.

Dems_533410
Related