protobuf marshaling of repeated empty bytes in golang

user_8582383 0
Viewed 1220

I have a simple protobuf message

//test.proto
syntax = "proto3";
package Bar;

option go_package = "/Bar";

message Foo {
    repeated bytes Foo = 1;
}

And the main program just takes an empty Foo message and marshals/unmarshals it:

package main

import (
    "fmt"

    "github.com/potuz/test/Bar"

    "google.golang.org/protobuf/proto"
)

func main() {
    msg := Bar.Foo{
        Foo: make([][]byte, 0),
    }
    buf, _ := proto.Marshal(&msg)

    newMsg := &Bar.Foo{}
    _ = proto.Unmarshal(buf, newMsg)

    if msg.Foo != nil {
        fmt.Println("msg.Foo is not nil")
    }

    if newMsg.Foo == nil {
        fmt.Println("newMsg.Foo is nil")
    }
}

The output of this program is

$ ./main
msg.Foo is not nil
newMsg.Foo is nil

Is there a clean way around this? I need to implement a grpc server/client where a message containing a repeated bytes field is present. And when the server responds with a non-nil message Foo = [][]byte{} the client gets nil and I need to deal with this particular case separately.

go protocol-buffers
1 Answers

tl;dr add a bool flag to the proto message to distinguish when the source slice is empty or nil.

message Foo {
    repeated ByteSlice slice = 1;
}

message ByteSlice {
    bool is_empty = 1;
    bytes payload = 2;
}

The semantics of a nil slice and a zero-length slice are the same, for the purposes of protobuffer serialization; repeated fields in Protobuffer wire format are represented with the "Length-delimited" type, which serializes to nothing if the input has zero length. And in Go both a nil and an empty slices have len(b) == 0.

The program below prints no output at all for the sample three messages:

proto

syntax = "proto3";

package test;

option go_package = ".;pb";

message Foo {
  repeated string a = 1;
}

message Bar {
  repeated bytes b = 1;
}

main

package main

import (
    "example.com/pb"
    "fmt"
    "google.golang.org/protobuf/proto"
)

func main() {
    foo := &pb.Foo{ A: make([]string, 0) } // empty string slice
    foobytes, _ := proto.Marshal(foo)
    fmt.Printf("%v\n", foobytes)

    bar1 := &pb.Bar{ B: nil } // nil 2D byte slice
    bar1bytes, _ := proto.Marshal(bar1)
    fmt.Printf("%v\n", bar1bytes)

    bar2 := &pb.Bar{ B: make([][]byte, 0) } // empty 2D byte slice
    bar2bytes, _ := proto.Marshal(bar2)
    fmt.Printf("%v\n", bar2bytes)

}

output (all empty):

[]
[]
[]

When the message is then deserialized into a struct, the byte field (repeated or not) will be missing, and it will result in a Go slice zero value, which is nil.

Protobuffer knows nothing of how the slice is represented in the source language, whether the slice was allocated and empty or just nil. The difference is specific to Go, not to protobuffers.

If you need to preserve this semantics, you can add a boolean field to the message:

message Foo {
    repeated ByteSlice slice = 1;
}

message ByteSlice {
    bool is_empty = 1;
    bytes payload = 2;
}

This will still unmarshal to a nil slice, but you'll be able to detect the empty state by checking the bool flag:

    newMsg := &Bar.Foo{}
    _ = proto.Unmarshal(buf, newMsg)
    if newMsg.Slice.IsEmpty {
        // restore the original state...
        newMsg.Slice.Payload = make([][]byte, 0)
        // ...or do something else knowing that the slice was empty
    }

Anyway remember that len(newMsg.Slice.Payload) will be 0 either way.

NOTE if what you have is a [][]byte that contains empty byte slices, this means the length is not zero, therefore it will marshal/unmarshal correctly:

    bar2 := &pb.Bar{
        B: [][]byte{
            make([]byte, 0), // or nil
            make([]byte, 0), // or nil
            make([]byte, 0), // or nil
        },
    }
    bar2bytes, _ := proto.Marshal(bar2)
    fmt.Printf("%v\n", bar2bytes)

output

[10 0 10 0 10 0]
user_41088030
Related