'Connect' multiple jobs by a common matrix value

Question

I'm using a matrix to run a template main.yml workflow that contains two jobs: get data and train model. I'm unsure if I'm using matrix strategy as intended (first time doing this).

The first job saves an artifact csv file and the second job uses this csv file.

I'm struggling to understand how to ensure that the corresponding matrix values apply across jobs.

(Aside, is it possible to define a matrix at the workflow level rather than job level?)

My template main.yml looks something like this:

on: [push]

jobs:
  ssql:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        GAME: ["FunGame", "AnotherFunGame"]
    name: Get data
    steps:
      - name: Checkout cum-rev repo
        uses: actions/checkout@v2 # Defaults to current repo - check out current repo
      - name: Checkout ds-ssql-gh-action
        uses: actions/checkout@v2
        with:
          repository: ourorg/ds-ssql-gh-action
          token: ${{ secrets.WORKFLOW_TOKEN }}
          ref: main
          path: './ds-ssql-gh-action'
      - name: Run query and save data
        uses: ./ds-ssql-gh-action
      - name: Upload data as artifact
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.GAME }}-data
          path: output_data/data.csv

  Rtrain:
    runs-on: ubuntu-latest
    name: Train model
    needs: ssql
    strategy:
      matrix:
        GAME: ["FunGame", "AnotherFunGame"]
    steps:
      - name: checkout current repo
        uses: actions/checkout@v2 # Defaults to current repo - check out current repo
      - name: Retreive data from ssql job 
        uses: actions/download-artifact@v2
        with:
          name: ${{ matrix.GAME }}-data
      - name: Just check csv file is there
        run: ls -l
      - name: Fit model and save as RDS
        uses: ./Rtrain # uses an action in this directory
      - name: Check the model is there
        run: ls -l
      - name: Upload model as artifact
        uses: actions/upload-artifact@v2
        with:
          name: ${{ matrix.GAME }}-Model
          path: ${{ matrix.GAME }}-Model.RDS

To ensure the right data are used in each job, I've used syntax to upload and download the artifact across the jobs as so:

  - name: Upload data as artifact
    uses: actions/upload-artifact@v2
    with:
      name: ${{ matrix.GAME }}-data
      path: output_data/data.csv

And

  - name: Retreive data from ssql job 
    uses: actions/download-artifact@v2
    with:
      name: ${{ matrix.GAME }}-data

So the name field is what I'm using to tell GH Actions which data to use with which. Is that the 'right' way?

Second, when I run this, the needs: ssql in the second job means that this job won't start till both jobs for GAME have run. But say the job ssql for GAME "FunGame" runs much quicker than that of "AnotherFunGame", is there a way I can tell GH actions that it only 'needs' the job corresponding to the correct game, as opposed to waiting for all jobs in ssql to complete?

I could just create two separate workflow ymls, one for FunGame and one for AnotherFunGame. But there are actually many more values of GAME and I would prefer to understand the 'right' approach to this.

How can I run a workflow yml as a template and pass arguments to each run, such as with matrix strategy and how can I ensure e.g.needs only corresponds to the job for specific GAME matrix value runs?

'Connect' multiple jobs by a common matrix value

0 Answers