gRPC request low throughput then errors

We have noticed that when using locust, we get much lower through put than using the ghz tool. We are using a custom grpc client so it is possible it is mis-configured, but we copied the example and just changed out our generated client.

With ghz, we can easily get to 800 RPS (OKs) while we can barely get to 100RPS (OKs) before it starts reporting timeouts and the failure rate becomes 100%. We have c5n.large instances as the workers (4vCPU) and have configured them to start 4 processed per node, one for each vCPU. I get the same characteristics if I run is locally so my belief is that it has to do with locust somehow.

Looking at cloudwatch, we can clearly see that the ghz tool traffic in the network logs, while when using locust, when the error occur, not traffic is being set to the LB's.

This maybe a little abstract, but I'm honestly stumped and want to know if others have encountered it and if someone has a resolution.

**** Updated *** Minor note, but to get similar through put I have to have 120 workers vs my local 16 core machine. Not apple to apples, I know, but still.

import json
from random import choice
import grpc
import protos.segments_pb2
import protos.segments_pb2_grpc
from client import client
from locust import events, User, task, run_single_user, constant, TaskSet, between
from locust.exception import LocustError
from locust.user.task import LOCUST_STATE_STOPPING
import gevent
import time
import grpc.experimental.gevent as grpc_gevent
import jwt
import time
import math
from locust import LoadTestShape

grpc_gevent.init_gevent()

def getMetadata():
    metadata = [('authorization', "aaaa")]
    return metadata

def credentials(context, callback):
    callback(getMetadata(), None)

def getCredentials():
    sslCred = grpc.ssl_channel_credentials()

    authcred = grpc.metadata_call_credentials(credentials)

    return grpc.composite_channel_credentials(sslCred, authcred)

class GrpcUser(User):
    #wait_time = between(2, 4)
    abstract = True

    stub_class = None

    def __init__(self, environment):
        super().__init__(environment)
        for attr_value, attr_name, in ((self.host, "host"), (self.stub_class, "stub_class")):
            if attr_value is None:
                raise LocustError(f"You must specify the {attr_name}.")
        self._channel = grpc.secure_channel(self.host, getCredentials())
        self._channel_closed = False
        stub = self.stub_class(self._channel)
        self.client = client.GrpcClient(environment, stub)

class SegmentsGrpcUser(GrpcUser):
    host = ""
    stub_class = protos.segments_pb2_grpc.MetadataStub

    @task
    def segmentWrite(self):
        if not self._channel_closed:
            meta_file = open("metadata.json")
            metadata = json.load(meta_file)
            data = choice(metadata)
            dataBytes = bytes(json.dumps(data), 'utf-8')
            self.client.Write(protos.segments_pb2.WriteRequest(ID=data['ID'], Metadata=dataBytes),timeout=10)
            meta_file.close()
        time.sleep(1)

class StepLoadShape(LoadTestShape):
    """
    A step load shape


    Keyword arguments:

        step_time -- Time between steps
        step_load -- User increase amount at each step
        spawn_rate -- Users to stop/start per second at every step
        time_limit -- Time limit in seconds

    """

    step_time = 10
    step_load = 50
    spawn_rate = 25
    time_limit = 600

    def tick(self):
        run_time = self.get_run_time()

        if run_time > self.time_limit:
            return None

        current_step = math.floor(run_time / self.step_time) + 1
        return (current_step * self.step_load, self.spawn_rate)