I have the following code, which creates a pandas.DataFrame with the timedelta between the dates in the index column and each column date. There are duplicate column names.
import pandas as pd
from datetime import datetime, timedelta
col = [3, 3, 4, 4, 5, 5]
col = [datetime(2021, 5, c) for c in col]
row = [1, 2, 3, 4, 5]
row = [datetime(2021, 5, r) for r in row]
df = pd.DataFrame(index=row, columns=col)
for col in df.columns:
df[col] = df.index - col
DataFrame output:
2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01 -2 days -2 days -3 days -3 days -4 days -4 days
2021-05-02 -1 days -1 days -2 days -2 days -3 days -3 days
2021-05-03 0 days 0 days -1 days -1 days -2 days -2 days
2021-05-04 1 days 1 days 0 days 0 days -1 days -1 days
2021-05-05 2 days 2 days 1 days 1 days 0 days 0 days
I also have this line of code, to replace the negative values with a large value:
df[df < timedelta(0)] = timedelta(999)
This gives me the following result in pandas~=1.3.0
2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01 999 days 999 days 999 days 999 days 999 days 999 days
2021-05-02 999 days 999 days 999 days 999 days 999 days 999 days
2021-05-03 0 days 0 days 999 days 999 days 999 days 999 days
2021-05-04 1 days 1 days 0 days 0 days 999 days 999 days
2021-05-05 2 days 2 days 1 days 1 days 0 days 0 days
However, in pandas~=1.4.0 I get this:
2021-05-03 2021-05-03 2021-05-04 2021-05-04 2021-05-05 2021-05-05
2021-05-01 999 days 999 days 999 days -3 days 999 days 999 days
2021-05-02 999 days 999 days 999 days -2 days 999 days 999 days
2021-05-03 999 days 999 days -1 days -1 days 999 days 999 days
2021-05-04 999 days 1 days 0 days 0 days 999 days -1 days
2021-05-05 999 days 2 days 1 days 1 days 999 days 0 days
There could be different things at play here:
- repeated column names
- column names which are
datetimeobjects - etc.
What has changed between pandas 1.3.x and 1.4.x to justify this change in behaviour? Is this something which pandas considers undefined behaviour? Or could it be a bug somewhere?
EDIT:
Using this line of code instead...
df = df.where(~(df < timedelta(0)), timedelta(999))
...to replace the negative values works. However, the question remains as to what could be causing the different behaviour between consecutive pandas versions.