Different behaviour between pandas 1.3.x and 1.4.x when handling datetime objects in DataFrame with repeated columns

Viewed 40

I have the following code, which creates a pandas.DataFrame with the timedelta between the dates in the index column and each column date. There are duplicate column names.

import pandas as pd
from datetime import datetime, timedelta

col = [3, 3, 4, 4, 5, 5]
col = [datetime(2021, 5, c) for c in col]
row = [1, 2, 3, 4, 5]
row = [datetime(2021, 5, r) for r in row]

df = pd.DataFrame(index=row, columns=col)

for col in df.columns:
  df[col] = df.index - col

DataFrame output:

           2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01    -2 days    -2 days    -3 days    -3 days    -4 days    -4 days
2021-05-02    -1 days    -1 days    -2 days    -2 days    -3 days    -3 days
2021-05-03     0 days     0 days    -1 days    -1 days    -2 days    -2 days
2021-05-04     1 days     1 days     0 days     0 days    -1 days    -1 days
2021-05-05     2 days     2 days     1 days     1 days     0 days     0 days

I also have this line of code, to replace the negative values with a large value:

df[df < timedelta(0)] = timedelta(999)

This gives me the following result in pandas~=1.3.0

           2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01   999 days   999 days   999 days   999 days   999 days   999 days
2021-05-02   999 days   999 days   999 days   999 days   999 days   999 days
2021-05-03     0 days     0 days   999 days   999 days   999 days   999 days
2021-05-04     1 days     1 days     0 days     0 days   999 days   999 days
2021-05-05     2 days     2 days     1 days     1 days     0 days     0 days

However, in pandas~=1.4.0 I get this:

           2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01   999 days   999 days   999 days    -3 days   999 days   999 days
2021-05-02   999 days   999 days   999 days    -2 days   999 days   999 days
2021-05-03   999 days   999 days    -1 days    -1 days   999 days   999 days
2021-05-04   999 days     1 days     0 days     0 days   999 days    -1 days
2021-05-05   999 days     2 days     1 days     1 days   999 days     0 days

There could be different things at play here:

  • repeated column names
  • column names which are datetime objects
  • etc.

What has changed between pandas 1.3.x and 1.4.x to justify this change in behaviour? Is this something which pandas considers undefined behaviour? Or could it be a bug somewhere?

EDIT:

Using this line of code instead...

df = df.where(~(df < timedelta(0)), timedelta(999))

...to replace the negative values works. However, the question remains as to what could be causing the different behaviour between consecutive pandas versions.

0 Answers
Related