I have implemented 2 services A and B, where A can talk to B via both gRPC (using grpc-node with Mali) and pure HTTP REST calls.

The request size is negligible.

The response size is 1000 items that look like this:

{
   "productId": "product-0",
   "description": "some-text",
   "price": {
      "currency": "GBP",
      "value": "12.99"
   },
   "createdAt": "2020-07-12T18:03:46.443Z"
}

Both A and B are deployed in GKE as services, and they communicate over the internal network using kube-proxy.

What I discovered is that the REST version is a lot faster than gRPC. The REST call's p99 sits at < 1s, and the gRPC's p99 can go over 30s.

Details

Node version and OS: node:14.7.0-alpine3.12

Dependencies:

    "google-protobuf": "^3.12.4",
    "grpc": "^1.24.3",
    "mali": "^0.21.0",

I have even created client-side-TCP-pooling by setting the gRPC option grpc.use_local_subchannel_pool=1, but this did not seem to help.

The problem seems to be the server side, as I can see that from the log that the grpc lib's call.startBatch call took many seconds to send data of size ~51kb. This is way slower than the REST version.

I also checked the CPU and network of the services are healthy. The REST version could send > 2mbps, whereas the gRPC version only manages ~150kbps.

Running netstat on service B (in gRPC) shows a number of ESTABLISHED TCP connections (as expected because of TCP pooling).

My suspicion is that the grpc-core C++ code is somehow less optimal than REST, but I have no proof.

Any ideas where I should look at next? Thanks for any helps

Update 1

Here're some benchmarks:

Setup

Blazemeter --REST--> services A --gRPC/REST--> service B

• request body (both lags) is negligible
• service A is a node service + Koa
• service B has 3 options:
  • grpc-node: node with grpc-node
  • gRPC + Go: Go implementation of the same gRPC service
  • REST + Koa: node with Koa
• Blazemeter --> service A: response payload is negligible, and the same for all tests
• serivce A --> service B: the gRPC/REST response payload is 1000 of ProductPrice:

message ProductPrice {
  string product_id = 1; // Hard coded to "product-x", x in [0 ... 999]
  string description = 2; // Hard coded to random string, length = 10
  Money price = 3;
  google.protobuf.Timestamp created_at = 4; // Hard coded
}

message Money {
  Currency currency = 1; // Hard coded to GBP
  string value = 2;  // Hard coded to "12.99"
}

enum Currency {
  CURRENCY_UNKNOWN = 0;
  GBP = 1;
}

The services are deployed to Kubernetes in GCP,

• instance type: n1-highcpu-4
• 5 pods each service
• 2 CPU, 1 GB memory each pod
• kube-proxy using cluster IP (not going via internet) (I've also test headless with clusterIP: None, which gave similar results)

Load

50rps

Results

service B using grpc-node

service B using Go gRPC

service B using REST with Koa

Network IO

Observations

• gRPC + Go is roughly on par with REST (I thought gRPC would be faster)
• grpc-node is 4x slower than REST
• Network isn't the bottleneck