I am trying to extract the "year" "month" "date" from the arrows timestamp[s] type. I know how to do it in pandas, as follows
import pyarrow.dataset as ds
dataset = ds.dataset(path, format="csv")
table = dataset.to_table()
## following codes wont work as table is arrow type and it doesn't have the map function.
table['date'] = table['time'].map(lambda t: pd.to_datetime(t, format="%Y-%m-%d %H:%M:%S"))
table['year'], table['month'], table['day'] = table['time'].apply(lambda x: x.year),table['date'].apply(lambda x: x.month), table['date'].apply(lambda x: x.day)
Easiest option is to convert this arrow DataFrame to pandas and do this. But the file size I am dealing with is much big and dont want to take the additional cost of converting it into pandas. As the end goal is after getting the year, month and day I want to use those as partitions in writing as parquet.
Trying my best to avoid the intermediate conversion, to make it much efficient in converting csv files to parquet. Is this achievable with the current functionalities available within arrow ?