Locust: How to use distributed mode programmatically

Question

I'm using locust to load test a bunch of apps, the problem is that it's using more than 90% of the CPU and so I want to switch to distributed mode with master and workers, I know how it's done in command line but I'm using locust as a library and it seems the docs don't cover this case, here's some snippets of my code:

User:

# host, port and reqs are external parameters
class User(HttpUser):
    host = f"{host}/{port}"

    @task
    def task1(self):
      for req in reqs:
        self.client.request(req['method'],req['path'],name=f"{port}",headers=req['headers'],data=req['body'])

    def on_start(self):
        self.client.verify = False

main:

env = Environment(user_classes=[User])
env.create_local_runner()

# env.create_web_ui("127.0.0.1", 8089)

gevent.spawn(stats_printer(env.stats))
gevent.spawn(stats_history, env.runner)

csvWriter = StatsCSVFileWriter(
            environment=env,
            base_filepath=f"{CWD}/outputs/{port}",
            full_history=True,
            percentiles_to_report=[0.5,0.6,0.7,0.8,0.9, 0.95, 0.99]
  )
gevent.spawn(csvWriter)
  
env.runner.start(100, spawn_rate=5)

gevent.spawn_later(30,lambda: saveReportAndQuit(env,port))

env.runner.greenlet.join()

# env.web_ui.stop()

python version: 3.9.7

Locust version: 2.4.0

user_7355637 · Answer

The Locust docs do in fact mention this. They don't have a full example, granted, but you should have what you need to do it from the docs.

The Environment instance’s create_local_runner, create_master_runner or create_worker_runner can then be used to start a Runner instance, which can be used to start a load test

create_worker_runner(master_host, master_port)

Create a WorkerRunner instance for this Environment

Parameters master_host – Host/IP of a running master node master_port – Port on master node to connect to

You'll need to know the master's IP address in code in order to use this, which Locust can't really help you with. But then you should be able to do something like this:

env = Environment(user_classes=[User])
env.create_worker_runner('0.0.0.0', 5557)
env.runner.greenlet.join()

I don't think you need to start the worker runner since the master should tell it to when your test starts, but I'm not 100% sure. Try it out and see what happens.

Also, if you're going to want to use the same file for master and workers, you'll need some logic to decide which to do.

Locust: How to use distributed mode programmatically

1 Answers