c++ write protobuf incrementally to file

user_1282931 0
Viewed 326

I have the following protobuf:

message FrameData {
    ... arbitrarily complex ...
}

message DataDump {
  optional uint64 headerField1 = 1;
  optional uint64 headerField2 = 2;
  optional uint64 headerField3 = 3;
  repeated FrameData frameData = 4;
}

I'm creating the frameData protobufs iteratively one by one and I want to write the DataDump protobuf to a file.

I can do that by simply building my DataDump protobuf and then calling SerializeToOstream - the problem with that is the data is a few 100gb and this way I would need to first build up all that data in ram before being able to dump it to disk.

Since the repeated field frameData is the last one in the protobuf, I assume it should be possible to first write the headers to the file and then stream out the elements of frameData iteratively after each FrameData protobuf has been generated.

How can this be done?

protocol-buffers c++
1 Answers

Protobuf messages (especially proto2 syntax messages) can be concatenated to merge their contents. Figuring how exactly that works requires reading the protobuf encoding format documentation.

In your case, you can first write DataDump message with the header fields into the file. After that, put whatever number of items you want in frameData and append that message to the end of the file. Repeat until all items have been written.

When a decoder reads the message, it appears as if all items are part of a big frameData array.

However, this can be a problem on the decoding side: in many implementations, this forces you to read the whole file into memory at once, instead of processing them in smaller chunks.

Especially if you want random access and some kind of indexing, storing data sets of hundreds of gigabytes could benefit from a more advanced storage format.

user_9147160
Related