How do I run multiple behave+python tests simultaneously without errors?

Question

I have a python web application that uses behave for behavioral testing. I have 5 *.feature files that each take a few minutes to run when I run them, both locally and on our Jenkins build server. I would like to run the five files in parallel rather than sequentially to save time. I can do this locally, but not on my build server. Here are the details:

Locally runs on Windows:

I can run all 5 files in separate command windows using these command:
- behave.exe --include "file_01.feature"
- behave.exe --include "file_02.feature"
- behave.exe --include "file_03.feature"
- behave.exe --include "file_04.feature"
- behave.exe --include "file_05.feature"
I can also run a python script that spins off 5 separate processes using the same command.
Both of these work, I have no problems

Build server runs on Linux:

When I try to run all five files using a similar command, some of the behave scenarios give me errors. The errors are one of these three:
- Message: unknown error: cannot determine loading status from disconnected: Unable to receive message from renderer
- Message: chrome not reachable
- Message: no such session
The behave scenarios that throw these errors seem to change with every test run.
Oddly, if I rearrange the 5 *.feature files into 3, it works. This is not an ideal solution though. Our application is growing. We'll have more feature files as it grows.

I suspect that there is some shared resource between the chrome drivers in the running behave tests, but I'm not sure. I can't explain why this works for me locally, but not on my build server. Nor can I explain why 3 files work, but not 5.

Has anyone seen errors like this when trying to run multiple behave tests simultaneously? Or do you know what I should be looking for? My project is big enough that it'd be difficult to put together a minimal example of my problem. That's why I haven't posted any code. I'm just wondering what I should be looking for, because I'm at a loss.

user_6537219 · Answer

This is how I running multiple features in parallel way.

from behave.__main__ import main as behave_main

@step(u'run in parallel "{feature}" "{scenario}"')
def step_impl(context, feature, scenario):
    t = threading.Thread(
        name='run test parallel',
        target=parallel_executor,
        args=[context, feature, scenario])
        #args=[context, 'parallel_actions.feature', 'Make Cab-Cab communication'])
    t.start()


def parallel_executor(context, feature_name, scenario):
    os.chdir(testenv.PARALLEACTIONS_PATH)
    behave_main('-i "{}" -n "{}" --no-capture --no-skipped'.format(feature_name, scenario))

And feature

Feature: testing parallel
  Scenario: parallel run
    When run in parallel "parallel_actions-1.feature" "Make Cab-Cab communication"
    And run in parallel "parallel_actions-1.feature" "Another Scenario"
    And run in parallel "another_parallel.feature" "Another Scenario 2"

I just create new thread and call behave executor directly, you don't need call behave.exe process 5 times separately but once. All features are executed at same time as parallel.

I cant answer your message errors, but you can try another approach (more behave way) to run behave features as parallel.

How do I run multiple behave+python tests simultaneously without errors?

1 Answers