How to get an exact representation of floats during `DataFrame.to_json`?

It refers to /pandas/io/json/_json.py codebase, by default the precision integer is upto 10, please see the below from the codebase..

def to_json(
    path_or_buf,
    obj,
    orient: Optional[str] = None,
    date_format: str = "epoch",
    double_precision: int = 10,
    force_ascii: bool = True,
    date_unit: str = "ms",
    default_handler: Optional[Callable[[Any], JSONSerializable]] = None,
    lines: bool = False,
    compression: Optional[str] = "infer",
    index: bool = True,
    indent: int = 0,

If you apply max precision then, you will be getting below..

>>> print(df.to_json(indent=2, orient='records', double_precision=15))
[
  {
    "0":0.0000000112345,
    "1":0.00000000112345,
    "2":0.000000000112345,
    "3":0.000000000011234,
    "4":0.000000000001123,
    "5":0.000000000000112,
    "6":0.000000000000011,
    "7":0.000000000000001,
    "8":1.12345e-16,
    "9":1.12345e-17,
    "10":1.12345e-18,
    "11":1.12345e-19,
    "12":1.12345e-20,
    "13":1.12345e-21,
    "14":1.12345e-22,
    "15":1.12345e-23,
    "16":1.12345e-24,
    "17":1.12345e-25,
    "18":1.12345e-26,
    "19":1.12345e-27,
    "20":1.12345e-28,
    "21":1.12345e-29,
    "22":1.12345e-30,
    "23":1.12345e-31,
    "24":1.12345e-32,
    "25":1.12345e-33,
    "26":1.12345e-34,
    "27":1.12345e-35,
    "28":1.12345e-36,
    "29":1.12345e-37,
    "30":1.12345e-38,
    "31":1.12345e-39
  }
]

Note: you will get value error if you use precision more than 15.

ValueError: Invalid value '20' for option 'double_precision', max is '15'

So, in a sense this is not same as json.dumps.

I'm not sure on achieving this with pd.DataFrame.to_json, but we can use pd.DataFrame.to_dict, json, and pd.read_json to achieve a full precision json representation from a pandas dataframe.

json_df = json.dumps(df.to_dict('index'), indent=2)
>>> print(json_df)
{
  "0": {
    "0": 1.12345e-08,
    "1": 1.12345e-09,
    "2": 1.12345e-10,
    "3": 1.12345e-11,
    "4": 1.12345e-12,
    "5": 1.12345e-13,
    "6": 1.12345e-14,
    "7": 1.12345e-15,
    "8": 1.12345e-16,
    "9": 1.12345e-17,
    "10": 1.12345e-18,
    "11": 1.12345e-19
  }
}

To read it back in, we can then do:

>>> pd.read_json(json_df, orient='index')
             0             1             2   ...            9             10            11
0  1.123450e-08  1.123450e-09  1.123450e-10  ...  1.123450e-17  1.123450e-18  1.123450e-19

[1 rows x 12 columns]

pd.DataFrame.to_json uses internal library pandas._libs.json and not a standard json module. This explains the difference in behavior. The former "normalizes" the numbers internally and does not expose API to control it. As a result, you have following options:

Either convert to dictionary and dump using standard json library (as mentioned earlier):

>>> print(json.dumps(df.to_dict(orient='records'), indent=2))

[
  {
    "0": 1.12345e-08,
    "1": 1.12345e-09,
    "2": 1.12345e-10,
    "3": 1.12345e-11,
    "4": 1.12345e-12,
    "5": 1.12345e-13,
    "6": 1.12345e-14,
    "7": 1.12345e-15,
    "8": 1.12345e-16,
    "9": 1.12345e-17,
    "10": 1.12345e-18,
    "11": 1.12345e-19
  }
]

This is perfectly legitimate solution.

You can use CSV format instead of JSON and specify desired floating point format:

>>> print(df.to_csv(float_format='%.10e', index=False))

0,1,2,3,4,5,6,7,8,9,10,11
1.1234500000e-08,1.1234500000e-09,1.1234500000e-10,1.1234500000e-11,1.1234500000e-12,1.1234500000e-13,1.1234500000e-14,1.1234500000e-15,1.1234500000e-16,1.1234500000e-17,1.1234500000e-18,1.1234500000e-19

Another option is to convert values to string before the "normalization" kicks in:

>>> print(df.astype(str).to_json(indent=2, orient='index'))

{
  "0":{
    "0":"1.12345e-08",
    "1":"1.12345e-09",
    "2":"1.12345e-10",
    "3":"1.12345e-11",
    "4":"1.12345e-12",
    "5":"1.12345e-13",
    "6":"1.12345e-14",
    "7":"1.12345e-15",
    "8":"1.12345e-16",
    "9":"1.12345e-17",
    "10":"1.12345e-18",
    "11":"1.12345e-19"
  }
}

The conversion to string will require special attention when reading back the JSON.

And finally, if you need the exact values, just use binary format such as parquet or pickle.

Firstly, you cannot get an "exact floating point representation" of your numbers, as they do not have an exact binary representation. E.g. decimal 1.2345e-8 can be represented exactly in 5 decimal digits (plus exponent) but when converted to binary, it's a repeating fraction so cannot be represented exactly in a finite number of binary digits. So there will inevitably be a rounding error. You can see this by printing to over full precision:

>>> [print(f'{eval(f"1.12345e-{i}"):.17e}') for i in range(8,20)]
1.12345000000000004e-08
1.12345000000000009e-09
1.12345000000000001e-10
1.12344999999999994e-11
1.12345000000000002e-12
1.12345000000000000e-13
1.12345000000000006e-14
1.12344999999999994e-15
1.12345000000000004e-16
1.12344999999999998e-17
1.12344999999999998e-18
1.12345000000000008e-19

That aside, I guess your problem is the pandas to_json implementation. What seems to be happening is that the precision (specified by double_precision) is a fixed number of decimal places (NOT significant figures), and it's just ignored if the value is lower than 1e-15.

I would view this as a bug, since any number between 1e-10 (the default value of double_precision) and 1e-15 will be completely lost - you'll just get zero.

Even if you use double_precision=15 you are getting increasingly inaccurate serialised values as you approach the limit, until suddenly they become more accurate again, so I'd suggest that only having the option of a fixed number of decimal places, which might be ignored, is questionable and there should at the very least be an option to serialise to a fixed number of significant figures, perhaps even by default to be consistent with the json module.

I would raise both of these as issues with the pandas developers.

As for a quick solution, go with @maneblusser's suggestion: use to_dict then json.dumps.