Use gRPC to share very large file

user_11708563 0
Viewed 8134

I want to use gRPC to share very large file (more than 6GB) between endpoints and a server.

The project where I'm currently working require a central server where endpoints can upload and download files. One of the constraint is that endpoints don't know each others, but they can receive and send messages each others from a common bus.

To implement this server and its communication with endpoints, I'm evaluating to use gRPC. Do you think is the best solution for file stream? what alternatives do I have?

thanks in advance.

grpc c#
2 Answers

If you really want to do this over gRPC, then the key thing is to make the response "server streaming", so that instead of returning 6GiB in one chunk, it returns multiple chunks of whatever size you need, for example maybe 128kiB at a time (or whatever); you can so this with something like:

syntax = "proto3";
message FileRequest {
    string id = 1; // or whatever
}
message FileResponse  {
    bytes chunk = 1; // some segment of the file
}

service SearchService {
  rpc GetFile(FileRequest) returns (stream FileResponse);
}

but nothing is automatic: it is now your job to write the multiple segments back.

I suspect a vanilla http download-style response may be simpler!

Marc_Gravell_233540

gRPC with client/server streaming is capable of handling upload/download of files. However, there's a discussion here on the performance of gRPC vs HTTP for file upload/download, which says HTTP is any day going to be faster to upload/download because this is just reading/writing incoming bytes, while gRPC performs additional serialization/deserialization for each message in the stream adding significant overhead.

There is another blog doing some benchmark on the same - https://ops.tips/blog/sending-files-via-grpc/ .

If you are looking to implement something that has to handle scale, you can do some more research.

user_51638070
Related