Exclude a combination involving a branch from GitHub Actions?

I'd like to exclude certain parts of my build matrix on certain branches. Something conceptually like the following, which doesn't work because branches is not a matrix variable.

name: Tests

on:
  pull_request:
  push:
    branches: [hackage, develop]

jobs:
  build:
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:

        os: [ubuntu-latest]
        resolver: [lts-3.22 , lts-17, lts-18, lts, nightly]
        exclude:
          - branches: [hackage]
            resolver: [nightly, lts]

Is there a way to do this idiomatically. Here, for example to exclude nightly and lts resolvers on the hackage branch?