I have a pandas data frame that is roughly 3 million rows and 5 columns. Each value is fairly small- roughly 20 characters or less. When I try to serialize into a protobuf file, it takes roughly an hour. Eventually, this process will have to serialize files that are 2000x longer, so I'm trying to make it more efficient.
I'm in python and can convert my dataframe into any other type of object to increase the overall speed. The only real thing that is set in stone is the output needs to be a protobuf.
So, my question is, is there a more efficient way to do this than adding each row one by one? What would that look like?
Here is the protofile:
syntax = "proto3";
package index_picker;
message unit{
string field1 = 1;
string field2= 2;
int64 field3 = 3;
string field4 = 4;
message field5{
int64 field51 = 1;
int64 field52 = 2;
}
repeated field5 field55= 5;
}
message row{
repeated unit unit1= 1;
}
I serialize by iterating through each row and adding:
book = row3_pb2.row()
for index, row in df.iterrows():
ProtoRow(book.unit1.add(), df, index)