Dataframe C or Fortran ordering in pandas

Question

I want to create pandas dataframes and be able to manipulate them in optimised code (numba).

Most optimised code will take series or dataframe inputs and store results in preallocated outputs.

@njit
def calc(u, v, w):
  for i in range (w.shape[0]):
    w[i] = some_f(u[i], v[i])

where some_f is a placeholder for operations that can be complex, with tests and loops, hence the use of numba. Most importantly, I want to avoid any useless copies of data in the process.

For vectorized functions such as the above, I want to use the same code for series and dataframes.

So for series u, v, w, I'll use:

calc(u.values, v.values, w.values)

For dataframes, I thought of reusing the same function with

calc(u.values.reshape(-1), v.values.reshape(-1), w.values.reshape(-1))

This only works if

The array ordering of the dataframe (C or Fortran) is consistent across the three dataframes
The reshape method is passed an argument order='C' or 'F' matching the original dataframe ordering, otherwise a copy is made.

Pandas does not seem to have a consistent policy for dataframe ordering.

For instance, a constructor

df=pandas.Dataframe(index=..., columns=..., data=0.0)

will return a C ordered array. While

df.copy()

will be Fortran ordered.

I wanted to know if some people here had encountered similar issues, and found a consistent way to ensure that dataframes are always in the same order (C or Fortran) without cluttering too much ordinary pandas code.

Dataframe C or Fortran ordering in pandas

0 Answers