Options for reducing Protocol Buffers's Message size

user_4050529 0
Viewed 294

I want to serialize object to data, and reduce the serialized data size as small as possible, and currently I'm testing Protocol Buffers.

I have read the official encoding doc, and did some optimizations for the varint numbers, apart from this, are there any other optimizations I can do to reduce the message size?

proto swift protocol-buffers java
1 Answers

Here are some suggestions, assuming you are using proto3 format.

Too much size optimisation may not be worth it - these all affect the generated code and/or require knowing the range / distribution of the field values in advance.

  • Use the smallest datatypes possible
    • Avoid writing negative integers as int32/int64, if a field will often be negative, encode as a sint32/sint64
    • Avoid fixed-length fields (fixed64, sfixed64, double, fixed32, sfixed32, float) if possible.
    • For varint-encoded types (int32, int64, uint32, uint64, sint32, sint64, bool, enum), the default value (0 or false) will omitted from the serialized data - use this to denote null/missing values if possible.
    • If a bool field is mostly true, invert the field's meaning to use the default value in more cases.
  • Minimised size/number of embedded messages
    • Avoid nesting of messages (e.g. "wrapper" types like Int32Value for nullable fields) where possible - these require an extra embedded message as well as the data.
    • Field tag numbers are written alongside data as varint - tags 1-15 occupy a byte, 16-2047 occupy 2 bytes. So reserve tags 1-15 for commonly used fields and give optional/rarely used fields higher tag numbers.
    • To visualize how much tag nesting your messages have, pipe the serialized data to protoc --decode_raw.
  • Compress messages after serialization if you can
user_9091990
Related