Including 3rd party protobuffer definition files

Artyom_66522 0
Viewed 127

I have a project that need to create/read protobuf files generated by other projects.

I want dlprimitives to be able to read files formatted as ONNX protobuf and Caffe protobuf

What is the best way to include them into project:

  1. Copy the files from original repo with readme reference to sources for updates
  2. Make caffe/onnx external sub-projects
  3. Download them on demand upon build

My thoughts:

  1. Is plain copy not sure how is it good for updating
  2. Creates huge subprojects and increases clone time for a single file, since it is impossible to have subproject of a signle file
  3. Assumes that build environment has internet access.

What would be better policy? How is it usually solved?

subproject code-sharing protocol-buffers
2 Answers

All of your approaches are valid.

  1. Protocol buffers is by design forward- or backward-compatible, so updating manually is fine. You'll can mitigate the issue by documenting the task or providing a script / task.

  2. External sub-projects is also fine, but you'll still have to update them manually (in the case of git submodules) and handle changed paths in the original project. You'll might want to consider contacting the projects and suggest splitting the format descriptions from the main project.

  3. Usually proxies or caches help in that case. Very few projects build nowadays without external dependencies.

In the end you should consider what you intended audience is. Maybe you can get some feedback from them how they would use your project?

user_12329280

ONNX and Caffe are the source-of-truth for the protos and I discourage you making copies of the protos (as the copies would be non-definitive).

To @eik's 2nd point, I think it's a good practice for protobuf maintainers to keep protos in distinct repos so that these repos are easier to integrate. I think this should be considered a best practice but it's rarely done. Occasionally protobuf maintainers generate multi-language sources on each proto change but this is just a fixed-cost saving and you may, of course, generate the SDK yourself at any point.

Neither of your referenced protos appears versioned distinctly from the repo. This should also be a best practice.

You should clone|recreate the 3rd-party proto repos on-demand for your builds, retaining the repo's reference and build clients for whichever languages you need. This keeps the proto contracts (potentially versioned) and sync'd with your (!) copies of generated clients.

Import the client (pacakges) using your language's package import. With Golang, for example, you can add replace in go.mod to redirect from e.g. https://github.com/onnx/onnx to your copy of the generated client.

user_6092900
Related