Does pandas current treatment of missing values have potential to cause serious problems in safety-critical software?

Viewed 51

I am new to Python and pandas and have just learned that pandas silently replaces NaN values with 0 when calculating sums, in contrast to explicit calculations as shown here:

import pandas as pd
import numpy  as np

np.NaN + np.NaN                              # Result: nan
pd.DataFrame([np.NaN,np.NaN]).sum().item()   # Result: 0.0

I think this has the potential to create hidden problems in data analyses by masking erroneous results, and am concerned about what the consequences may be in safety-critical or high-consequence applications. Furthermore, NaN has a specific meaning in the IEEE 754 standard to indicate indeterminate results, not the absence of a value.

I appreciate that pandas' Descriptive Statistics methods have a skipna argument. However, skipna is by default True, thereby masking the presence of missing values to casual users and novice programmers... and perhaps others.

Does this have the potential to cause problems in safety-critical software or high-stakes data analyses?

0 Answers
Related