How to extract percentile values using Xarray?

Omi 0
Viewed 31

I'm using xarray to extract data from a NetCDF file (.nc). I need to extract the 75th percentile value for one of the variables (concentration of chemical) at each time step across a set of dimensions (latitude, longitude, depth). I'm using the below code to do this

df = xr.open_dataset("10y_125_365_concentration.nc") 
c_75 = df.concentration.quantile(0.75, dim=('latitude', 'longitude', 'depth'))

The result gives '0' as output for first few time steps before giving specific values of concentration. It appears that the code calculates 75th percentile values across entire array, however, I need to extract 75 percentile values excluding the zeros in the array (the length of the concentration array with values other than 0 changes with time and the number of zero values decreases as time step increases)

python-xarray python
1 Answers

xr.DataArray.quantile has an optional argument skipna which defaults to True for float data types. So by default, if df.concentration has float data, xarray will use np.nanpercentile under the hood and will skip invalid data.

So, all you need to do is tell xarray that 0s are invalid and should be skipped. You can do this with DataArray.where, which will return np.nan anywhere the condition is False:

c_75 = df.concentration.where(df.concentration != 0).quantile(
    0.75, dim=('latitude', 'longitude', 'depth')
)

Performance note: currently, np.nanpercentile is much, much slower than np.percentile when operating over a subset of the axes of a high-dimensional array (e.g. most xarray use cases). This seems unavoidable in your case, but in other cases where you do not have invalid values in your array, it's best to pass skipna=False to allow xarray to use the much faster np.percentile operation under the hood. Careful with this though - np.percentile returns incorrect results if NaNs are present.

Michael Delgado0
Related