I have large dataframes in CSV format, each file is about 5 - 6 GB. The large dataframe has a column named Unix_Timestamp, I need to split the large dataframes to a set of smaller dataframes by the timestamp values. Please see the diagram below:
The simple solution would be to parse the CSV file line by line and use a dictionary to organize those rows with the same timestamp values, then write the rows with same timestamp to a CSV file.
But I am also trying to use Dask or the regular Pandas library to process it with higher efficiency. I am not sure if the GroupBy would work for splitting the large dataframe.
I have code below to create an example dataframe that I would like to split
from collections import OrderedDict
import pandas as pd
table = OrderedDict()
col_names = []
num_rows = 10 # number of rows for each timestamp (ts1, ts2)
ts1 = [1641220210] * num_rows
ts2 = [1851220221] * num_rows
timestamps = ts1 + ts2 # concatenate
all_values = []
# print(timestamps)
for i in range(4):
col_names.append(f'Col{i+1}')
values = [(i+1) * 100] * (num_rows * 2)
all_values.append(values)
table['Unix_Timestamp'] = timestamps
for i in range(len(col_names)):
table[col_names[i]] = all_values[i]
df = pd.DataFrame(table)
df
I am using this simple for-loop to split the large dataframe, but wondering if there's a more efficient way to do the split on large dataframes:
df_list = [d for _, d in df.groupby(['Unix_Timestamp'])]

