execute multiple steps in parallel in CircleCI

Question

I have a CircleCI job with the following structure.

jobs:
  test:
    steps:
      - checkout
      - run #1
        ...<>
      - run #2
        ...<>
      - run #3
        ...<> 
      - run #4
        ...<>

I want to execute step #1 first, and then steps #2-4 in parallel.
#1, #2, #3, and #4 take around ~4 min., ~1 min., ~1 min., and ~1 min., respectively.

I tried to split the steps to different jobs and use workspaces to pass the installed artifacts from #1 to #2-4. However, because of the large size of the artifacts, it took around ~2 min. to persist & attach workspace, so the advantage of splitting jobs was cancelled out.

Is there a smart way to run #2-4 in parallel without significant overhead?

user_7748346 · Answer

If you want to run the commands in parallel, you need to move these commands into a new job, otherwise, CircleCI will follow the structure of your step, running the commands only when the last one is finished. Let me give you an example. I created a basic configuration with 4 jobs.

- npm install
- test1 (that will run at the same time as test2) but only when the npm install finish
- test2 (that will run at the same time as test1) but only when the npm install finish
- deploy (that will only run after the 2 tests are done)

Basically, you need to split the commands between jobs and set a dependency from what you want.

See my config file:

version: 2.1

jobs:
  install_deps:
    docker:
      - image: circleci/node:14

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      
      - run: echo "running npm install"
      - run: npm install
      - persist_to_workspace:
          root: .
          paths:
            - '*'
  test1:
    docker:
      - image: circleci/node:14

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - attach_workspace:
            at: .

      - run: echo "running the first test and also will run the test2 in parallel"
      - run: npm test

  test2:
    docker:
      - image: circleci/node:14

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - attach_workspace:
            at: .

      - run: echo "running the second test in parallel with the first test1"
      - run: npm test
  deploy:
    docker:
      - image: circleci/node:14

    steps:
      - checkout
      - setup_remote_docker:
          docker_layer_caching: true
      - attach_workspace:
            at: .

      - run: echo "running the deploy job only when the test1 and test2 are finished"
      - run: npm run build

# Orchestrate our job run sequence
workflows:
  test_and_deploy:
    jobs:
      - install_deps
      - test1:
          requires:
             - install_deps
      - test2:
          requires:
             - install_deps
      - deploy:
          requires:
             - test1
             - test2

Now see the logic above, the install_dep will run with no dependency, but the test1 and the test2 will not run until the install_dep is finished.

Also, the deploy will not run until both tests are finished.

I've run this config, in the first image we can see that the other jobs are waiting for the first one to finish, in the second image we can see both tests are running in parallel and the deploy job is waiting for them to finishes. In the third image, we can see that the deploy job is running.

execute multiple steps in parallel in CircleCI

1 Answers