I am running a performance test on my unary gRPC server using GHZ, under load about 100K request the response time is high(about 1s to 1.5s) for initial few thousand requests. After some debugging what I am observing is that there is a delay of about 500ms between the end of gRPC server interceptor and invocation of the gRPC service method, and about 500ms delay between the end of service method and invocation of SimpleForwardingServerCallListener.onComplete. What could be causing this delay, I have configured 8 threads for netty server and 8 threads for gRPC cancellation context executor.
1 Answers
Had posted the same question on github java-grpc, below is the thread. https://github.com/grpc/grpc-java/issues/7372
TL;DR
A large number of concurrent users(num users >> num gRPC worker threads ) would mean the system is oversubscribed and unable to process the requests, thus creating a backlog of requests to be processed. This causes delay in processing the request even though the request is accepted by the gRPC event loop thread. For example. Total time for processing a single request= 375ms Time between end of gRPC server interceptor(ServerInterceptor.interceptCall) and invocation of gRPC service = 162ms Time to process the request by gRPC service = 65ms Time between end of gRPC service and invocation of SimpleForwardingServerCallListener.onComplete = 148ms
382/61=6.2, so the system is 6.2x oversubscribed. 100/6.2=16 which would be closer to the maximum concurrent RPCs it can handle.
Related