I have a data set that is supplied via CSV. Whenever that csv is supplied in a leap year it contains the 29th of February and whenever it is supplied in a non-leap year it does not. I only get a single year of data and I have to copy that data forward for X amount of years.
I need to remove the 29th if it exists in years it shouldn't and create it if it doesn't exist in the years it should.
EDIT:
I am going back to the beginning so that the entire picture is presented.
I am given 3 csv files that I utilize to make a single dataframe:
- df0 contains 4 columns that I require to filter and create a final dataframe from df1
- df2 contains 3 columns that I require to filter and create a final dataframe from df1
- df1 is the dataframe that contains a single year of data (but it is initially missing the year, I have to add it from df0 based on start date)
df0 Partitioned for the correct columns looks like this:
+------------+------------+--------------------+--------------------+
| project_id | start_date | degredation_factor | snapshot |
+------------+------------+--------------------+--------------------+
| pid1 | 1/1/2021 | 0.60% | 2/28/2020 18:35:46 |
| pid2 | 1/1/2024 | 0.40% | 2/28/2020 18:35:46 |
+------------+------------+--------------------+--------------------+
*note this could have 20+ unique projects in it
df2 looks like this :
+------------+---------------+--------------------+
| project_id | duration_year | snapshot |
+------------+---------------+--------------------+
| pid1 | 10 | 2/28/2020 18:35:46 |
| pid1 | 15 | 2/28/2020 18:35:46 |
| pid2 | 20 | 2/28/2020 18:35:46 |
| pid2 | 25 | 2/28/2020 18:35:46 |
+------------+---------------+--------------------+
*can contain multiple lines per project, need only the line with the longest duration
df1 looks like this:
+-----+-------+------+-------------+--------------------+------------+------+
| day | month | hour | hourly_rate | snapshot | project_id | year |
+-----+-------+------+-------------+--------------------+------------+------+
| 1 | 1 | 1 | 123.43 | 2/28/2020 18:35:46 | pid1 | 2021 |
| 1 | 1 | 2 | 120.11 | 2/28/2020 18:35:46 | pid1 | 2021 |
| ... | ... | ... | ... | ... | ... | ... |
| 31 | 12 | 24 | 123.43 | 2/28/2020 18:35:46 | pid1 | 2021 |
+-----+-------+------+-------------+--------------------+------------+------+
*every day, every hour, 1 year
I have to take that 1 year dataframe and append to it all the forward years for every project. so in the case of pid1 I need to append 2022 - 2036 and I need to perform the degradation on the hourly rate for 2022 and beyond ( formula : hourly_rate * (1 - (float(row.loc['degradation_factor_solar'].strip('%')) * (year# - 1)/ 100))
my code previously (which was doing all of the appending and calculations, except it had the 29th of February in all years and hence the problem):
df0_partition_1 = df0[['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']]
df0_partition_2 = df0_partition_1.groupby(
['project_id', 'start_date', 'degradation_factor', 'snapshot_datetime']).size().reset_index()
df2_partition_1 = df2.groupby(['project_id', 'snapshot_datetime'])['duration_year'].max().reset_index()
df_merge = pd.merge(df0_partition_2, df2_partition_1, on=['project_id', 'snapshot_datetime'], how='left')
df_parts = df_merge[
['project_id', 'start_date', 'duration_year', 'degradation_factor', 'snapshot_datetime']].dropna()
for index, row in df_parts.iterrows():
df1_filtered = df1[(df1['project_id'] == row['project_id']) &
(df1['snapshot_datetime'] == row['snapshot_datetime'])]
df1_filtered['year'] = pd.to_datetime(row['start_date']).year
df1_filtered.reset_index(inplace=True, drop=True)
df1_filtered.drop(columns='project_name', inplace=True)
df_stg_1 = df1_filtered.copy() # deep=True)
df_stg_2 = pd.DataFrame()
df_final = pd.DataFrame()
for y in range(1, int(row['duration_year']) + 1):
year = df1_filtered['year'] + (y - 1)
hourly_production = df1_filtered['hourly_production']
df_stg_1['year'] = year
df_stg_1['hourly_production'] = hourly_production * (
1 - (float(row.loc['degradation_factor_solar'].strip('%')) * (y - 1)/ 100))
df_stg_2 = df_stg_2.append(df_stg_1)
df_final = df1_filtered.append(df_stg_2)
this was giving me all of the years, but it presented the leap year problem when the data was input on a leap year. The 29th exists and it gets populated into every year. On Non-leap years when they provide the data it will lack the leap day and thus the leap day would have to be created from the 28th.
This is now a partial duplication of a previous post, my apologies. I needed to expand to show the entire issue and why I am trying to solve what I am trying to solve.
I am happy to create the final dataframe in a different manner if it gets rid of this issue for leap day.
EDIT 2:
I tried df_stg_3.drop(df_stg_3[(df_stg_3['year'] % 4 != 0) & (df_stg_3['month'] == 2) & (df_stg_3['day'] == 29)].index)
for the drop is not actually working, I had the wrong file.