Append date column to parquet file

Question

I am very new to python and programming in general. I have a job where I have received a bunch of data that I need to transform. The data is in parquet format and the initial request was to compile all the different files into one csv. I did this by first manually extracting the files from their relevant folders and then running this script:

import dask.dataframe as dd
ddf = dd.read_parquet('C:/Temp/Automation/2020/*.parquet')
ddf.to_csv("df_all.csv", 
           single_file=True, 
           index=False
)

This worked well but there is now a request to add the relevant month for each parquet file to the final CSV. The data has been received like this:

[Folder Week View][1]

And this is the contents of each folder:

[Week 27 contents][2]

So my question is in 2 parts I guess:

How do I run a script to pick the parquet files out of the folders without having to manually extract them
Add the relevant month to each parquet file before compiling them all to a single CSV [1]: https://i.stack.imgur.com/WdfkJ.png [2]: https://i.stack.imgur.com/7lnTJ.png

As the file is in a folder named by week of the year, would it be possible to either add that week number to a column or somehow know that week 27 is June?

Thanks, David

Append date column to parquet file

0 Answers