Handling Leap Day, both when it exists and when it doesn't

Viewed 141

I have a data set that is supplied via CSV. Whenever that csv is supplied in a leap year it contains the 29th of February and whenever it is supplied in a non-leap year it does not. I only get a single year of data and I have to copy that data forward for X amount of years.

I need to remove the 29th if it exists in years it shouldn't and create it if it doesn't exist in the years it should.

EDIT:

I am going back to the beginning so that the entire picture is presented.

I am given 3 csv files that I utilize to make a single dataframe:

  1. df0 contains 4 columns that I require to filter and create a final dataframe from df1
  2. df2 contains 3 columns that I require to filter and create a final dataframe from df1
  3. df1 is the dataframe that contains a single year of data (but it is initially missing the year, I have to add it from df0 based on start date)

df0 Partitioned for the correct columns looks like this:

+------------+------------+--------------------+--------------------+
| project_id | start_date | degredation_factor |      snapshot      |
+------------+------------+--------------------+--------------------+
| pid1       | 1/1/2021   | 0.60%              | 2/28/2020 18:35:46 |
| pid2       | 1/1/2024   | 0.40%              | 2/28/2020 18:35:46 |
+------------+------------+--------------------+--------------------+

*note this could have 20+ unique projects in it

df2 looks like this :

+------------+---------------+--------------------+
| project_id | duration_year |      snapshot      |
+------------+---------------+--------------------+
| pid1       |            10 | 2/28/2020 18:35:46 |
| pid1       |            15 | 2/28/2020 18:35:46 |
| pid2       |            20 | 2/28/2020 18:35:46 |
| pid2       |            25 | 2/28/2020 18:35:46 |
+------------+---------------+--------------------+

*can contain multiple lines per project, need only the line with the longest duration

df1 looks like this:

+-----+-------+------+-------------+--------------------+------------+------+
| day | month | hour | hourly_rate |      snapshot      | project_id | year |
+-----+-------+------+-------------+--------------------+------------+------+
|   1 |     1 |    1 |      123.43 | 2/28/2020 18:35:46 | pid1       | 2021 |
|   1 |     1 |    2 |      120.11 | 2/28/2020 18:35:46 | pid1       | 2021 |
| ... |   ... |  ... |         ... | ...                | ...        |  ... |
|  31 |    12 |   24 |      123.43 | 2/28/2020 18:35:46 | pid1       | 2021 |
+-----+-------+------+-------------+--------------------+------------+------+

*every day, every hour, 1 year

I have to take that 1 year dataframe and append to it all the forward years for every project. so in the case of pid1 I need to append 2022 - 2036 and I need to perform the degradation on the hourly rate for 2022 and beyond ( formula : hourly_rate * (1 - (float(row.loc['degradation_factor_solar'].strip('%')) * (year# - 1)/ 100))

my code previously (which was doing all of the appending and calculations, except it had the 29th of February in all years and hence the problem):

        df0_partition_1 = df0[['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']]
        df0_partition_2 = df0_partition_1.groupby(
            ['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']).size().reset_index()
        df2_partition_1 = df2.groupby(['project_id', 'snapshot_datetime'])['duration_year'].max().reset_index()
        df_merge = pd.merge(df0_partition_2, df2_partition_1, on=['project_id', 'snapshot_datetime'], how='left')
        df_parts = df_merge[
            ['project_id', 'start_date', 'duration_year', 'degradation_factor', 'snapshot_datetime']].dropna()

        for index, row in df_parts.iterrows():
            df1_filtered = df1[(df1['project_id'] == row['project_id']) &
                               (df1['snapshot_datetime'] == row['snapshot_datetime'])]
           df1_filtered['year'] = pd.to_datetime(row['start_date']).year
            df1_filtered.reset_index(inplace=True, drop=True)
            df1_filtered.drop(columns='project_name', inplace=True)
            df_stg_1 = df1_filtered.copy()  # deep=True)
            df_stg_2 = pd.DataFrame()
            df_final = pd.DataFrame()

            for y in range(1, int(row['duration_year']) + 1):
                year = df1_filtered['year'] + (y - 1)
                hourly_production = df1_filtered['hourly_production']
                df_stg_1['year'] = year
                df_stg_1['hourly_production'] = hourly_production * (
                        1 - (float(row.loc['degradation_factor_solar'].strip('%')) * (y - 1)/ 100))
                df_stg_2 = df_stg_2.append(df_stg_1)
            df_final = df1_filtered.append(df_stg_2)

this was giving me all of the years, but it presented the leap year problem when the data was input on a leap year. The 29th exists and it gets populated into every year. On Non-leap years when they provide the data it will lack the leap day and thus the leap day would have to be created from the 28th.

This is now a partial duplication of a previous post, my apologies. I needed to expand to show the entire issue and why I am trying to solve what I am trying to solve.

I am happy to create the final dataframe in a different manner if it gets rid of this issue for leap day.

EDIT 2:

I tried df_stg_3.drop(df_stg_3[(df_stg_3['year'] % 4 != 0) & (df_stg_3['month'] == 2) & (df_stg_3['day'] == 29)].index)

for the drop is not actually working, I had the wrong file.

2 Answers

This is an idea on merging on day and month:

def move_data(df, future):
    new_df = pd.DataFrame({'date':pd.date_range(f'{future}/01/01',
                                                   f'{future}/12/31', 
                                                   freq='D')
                             })

    new_df['day'] = new_df['date'].dt.day
    new_df['month'] = new_df['date'].dt.month

    new_df = (new_df.merge(df, on=['day','month'],how='left')
                    .assign(year=future)
                    .drop('date', axis=1)
               )
    return new_df

Data

   month  day  hour snapshot_datetime  year
0      2   28     1   2/28/2020 16:51  2020
1      2   28     2   2/28/2020 16:51  2020
2      2   28     3   2/28/2020 16:51  2020
3      2   29     1   2/28/2020 16:51  2020
4      2   29     2   2/28/2020 16:51  2020
5      2   29     3   2/28/2020 16:51  2020

Output

move_data(df, 2021).dropna()

    day  month  hour snapshot_datetime  year
58   28      2   1.0   2/28/2020 16:51  2021
59   28      2   2.0   2/28/2020 16:51  2021
60   28      2   3.0   2/28/2020 16:51  2021

and

move_data(df, 2024).dropna()

    day  month  hour snapshot_datetime  year
58   28      2   1.0   2/28/2020 16:51  2024
59   28      2   2.0   2/28/2020 16:51  2024
60   28      2   3.0   2/28/2020 16:51  2024
61   29      2   1.0   2/28/2020 16:51  2024
62   29      2   2.0   2/28/2020 16:51  2024
63   29      2   3.0   2/28/2020 16:51  2024

Note you don't need dropna if you have data for everyday in your data.

Here is where I ended up. It may not be the most elegant, but it does work.

        df0_partition_1 = df0[['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']]
        df0_partition_2 = df0_partition_1.groupby(
            ['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']).size().reset_index()
        df2_partition_1 = df2.groupby(['project_id', 'snapshot_datetime'])['duration_year'].max().reset_index()
        df_merge = pd.merge(df0_partition_2, df2_partition_1, on=['project_id', 'snapshot_datetime'], how='left')
        df_parts = df_merge[
            ['project_id', 'start_date', 'duration_year', 'degradation_factor_solar', 'snapshot_datetime']].dropna()

        for index, row in df_parts.iterrows():
            df1_filtered = df1[(df1['project_id'] == row['project_id']) &
                               (df1['snapshot_datetime'] == row['snapshot_datetime'])]
            dest_key_name = 'dw/project_8760/{}/{}/{}'.format(row['project_id'], file_partition_time, 'df.csv')

            df1_filtered['year'] = pd.to_datetime(row['start_date']).year
            df1_filtered.reset_index(inplace=True, drop=True)
            df1_filtered.drop(columns='project_name', inplace=True)
            df_stg_1 = df1_filtered.copy()
            df_stg_2 = pd.DataFrame()
            df_final = pd.DataFrame()

            for y in range(2, int(row['duration_year']) + 1):
                year = df1_filtered['year'] + (y - 1)
                hourly_production = df1_filtered['hourly_production']
                df_stg_1['year'] = year
                df_stg_1['hourly_production'] = hourly_production * (
                        1 - (float(row.loc['degradation_factor'].strip('%')) * (y - 1)/ 100))
                df_stg_2 = df_stg_2.append(df_stg_1)
            df_stg_3 = df1_filtered.append(df_stg_2)

            if df1_filtered[(df1_filtered['month'] == 2) & (df1_filtered['day'] == 29)].empty is False :
                filter_final_1 = df_stg_3[(df_stg_3.year % 4 != 0) & (df_stg_3.month == 2)]
                filter_final_2 = filter_final_1[(filter_final_1.day == 29)]
                df_final = pd.concat([df_stg_3, filter_final_2, filter_final_2]).drop_duplicates(keep=False)
            else:
                df_stg_3_filtered = df_stg_3[(df_stg_3.year % 4 == 0) & (df_stg_3.month == 2)]
                df_stg_4 = df_stg_3_filtered[(df_stg_3_filtered.day == 28)]
                df_stg_4['day'] = 29
                df_final = pd.concat([df_stg_3, df_stg_4])
                df_final.sort_values(by=['year', 'month', 'day', 'hour'])
Related