Issues with ssh for github actions with private submodules

I have a cmake action configured, for a c++ repo with some private submodules. The workflow is

name: CMake

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  BUILD_TYPE: Release

jobs:
  build:
    runs-on: ubuntu-latest
    container: user/image:latest

    steps:
    - uses: actions/checkout@v2
      with:
         submodules: true

    - name: Configure CMake
      with:
          run: cmake -B ${{github.workspace}}/build -DCMAKE_BUILD_TYPE=${{env.BUILD_TYPE}}

    - name: Build
      # Build your program with the given configuration
      run: cmake --build ${{github.workspace}}/build --config ${{env.BUILD_TYPE}}

    - name: Test
      working-directory: ${{github.workspace}}/build
      run: ctest -C ${{env.BUILD_TYPE}}   

But it fails to clone private submodules:

  remote: Repository not found.
  Error: fatal: repository 'https://github.com/user/submodule.git/' not found
  Error: fatal: clone of 'git@github.com:user/submodule.git' into submodule path '/__w/folder/external/submodule' failed

I tried using a SSH_PRIVATE_KEY as a secret for the private half of the ssh keypair. And the public key was added to the github user settings.

But then not even the main repo is being able to get cloned:

Load key "/__w/_temp/5103ef84-9e3f-4dbf-b26d-a8a7d959bbb1": invalid format
git@github.com: Permission denied (publickey).
Error: fatal: Could not read from remote repository.

I also tried to create a Personal Access Token for the docker build and pass it via a "PAT" secret:

    steps:
    - uses: actions/checkout@v2
      with:
        token: ${{ secrets.PAT}}

With the PAT variable, the error becomes:

Error: Input required and not supplied: token

Adding a ssh-known-hosts: 'github.com' also did not work.

Does anyone know what else can I try?