Protobuf: trying to add 'scalingfactor' as metadata to .proto file

user_7006123 0
Viewed 212

I am trying to (mis??)use the .proto definition to add a scalingFactor to all my scalar values such that both sides that are communicating can decide for themselves to internally use money values in cents, floats or decimals. In that I want to end up with just an integer going over the wire. Not two integers (value and scalingFactor for instance, or value and nanos).

If I had the freedom to just add some metadata at field level by putting it between [], it would look like this:

    syntax = "not valid proto3....";
    
    message Product1 {
      int32 id = 1;
      string productName = 2;
      int32 priceInCents = 3
      int32 weightInTons = 4
    }
    message Product2 {
      int32 id = 1;
      string productName = 2;
      int32 price = 3 [scalingFactor = .01]
      int32 weight = 4 [scalingFactor = 1000]
    }

I would like to end up with a definition like it is described (but not valid of course) for Product2. Such that both sides can get the scalingFactor from the .proto definition. And then use that scalingFactor in their own code. Some might convert the price to a decimal in JavaScript or leave it in cents in C (e.g. a struct called price with int32 for value (in cents) and int32 for scale (e.g. -2 representing .01) )

Is there an obvious way to do this? Ideally it is simple.

UPDATE based on answer:

I now realise by paxdiablo's reaction that I was both unclear and maybe even in doubt myself.

Unclear part: If price is $2.50 and weight is 7000kg, I want to transmit 250 and 7 over the wire in all cases. What I was not looking for is for protobuf to handle the scaling for me. It is more that I wanted to have the context of how transmitted values are scaled to become available to the code.

Doubt: I am not sure I want to use the generated classes in my own code at both sides. I saw protobuf more as the transport mechanism only. And create my own variables again based on the transmitted values or values to be transmitted.

protocol-buffers
2 Answers

It looks like you want to set weight to 7 in your code, and then have protobuf magically insert 7000 in the on-the-wire message, based on a scale specified in the proto file.

If that's the case, I concur, this looks very much like a misuse :-)

Unfortunately, protobuf doesn't do that. You either have to scale the value yourself so that the correct value goes over the wire to the other side, or add another field with the scaling so the other side can fix it.

Since you state you don't want to do the latter, you'll have to do the former.

Otherwise there's no way for both ends to agree on the scale.

If, on the other hand, what you want is something in the protobuf file that just specified the scale, you can use an enum for that, something like:

// Scale factors to use, extract either MUL or DIV, one needs 
// to be zero, the other is an actual multiplier/divisor.

enum PriceScale {
    PRICE_SCALE_DIV = 0;
    PRICE_SCALE_MUL = 100;     // 2.50 goes over wire as 250.
}
enum WeightScale {
    WEIGHT_SCALE_MUL = 0;
    WEIGHT_SCALE_DIV = 1000;   // 7000 goes over wire as 7.
}
message Product2 {
  int32 id = 1;
  string product_name = 2;
  int32 price = 3;
  int32 weight = 4;
}

but you'll still have to manually scale the values sent over the wire, either at the sending or receiving end. It's also probably not the intended use case for enumerations. We use this trick for storing constants in the proto files so that our code can get access to it (specifically, some strings have length limits we want to enforce).

For actually scaling, you would have to do something like this at the sending side:

// Should be done once early on, and stored somewhere.

auto priceScaler = (PriceScale::PRICE_SCALE_DIV == 0)
    ? 1.0 * PriceScale::PRICE_SCALE_MUL
    : 1.0 / PriceScale::PRICE_SCALE_DIV
auto weightScaler = (WeightScale::WEIGHT_SCALE_DIV == 0)
    ? 1.0 * WeightScale::WEIGHT_SCALE_MUL
    : 1.0 / WeightScale::WEIGHT_SCALE_DIV

// Construct message.

protoMsg.set_id(42);
protoMsg.set_product_name("sprocket");
protoMsg.set_price(static_cast<uint32_t>(actualPrice * priceScaler));
protoMsg.set_weight(static_cast<uint32_t>(actualWeight * weightScaler));

And, when extracting the values:

// As with sender, scaler variables should have been created.

auto actualPrice = protoMsg.price() / priceScaler;
auto actualWeight = static_cast<uint32_t>(protoMsg.weight() / weightScaler);
user_148600

Adding custom metadata to fields and messages is reasonably common and can be done with field options. They would look something like this:

# scaling_factor.proto
syntax = "proto2";
import "google/protobuf/descriptor.proto";

package my_options;

extend .google.protobuf.FieldOptions {
  optional float scaling_factor = 12345;
}


# product.proto
syntax = "proto3";
import "scaling_factor.proto";

message Product2 {
  (...)
  int32 price = 3 [(my_options.scaling_factor) = .01];
  int32 weight = 4 [(my_options.scaling_factor) = 1000];
}

There are APIs in each language that let you get the options for specific fields. Note that the field extension must defined in a proto2 file (proto3 doesn't have extensions).

However, you should be aware that this is strictly metadata and has no effect on how the messages are serialized. A compiled program will see the value of the annotation from when it was built. If you have two binaries that are using this proto definition, they will use whatever value of the annotations that existed when you compiled each binary - it would be very complicated to change a scaling factor like this, since it changes the semantics of a field. That may or may not be an issue for you.

user_3408620
Related