How do I manage secrets in docker-compose.yml regarding .ssh/id_rsa?

In a Python project I need to pass my private SSH key (~/.ssh/id_rsa) into the container securely because of the private dependencies. So I decided to use docker secrets. I have my Dockerfile like this:

# Base image
FROM python:3.9-slim

# Set working directory
WORKDIR /app

# Install the dependencies: openssh-client, git
RUN apt-get update && \
    apt-get install -y openssh-client && \
    apt-get install -y --no-install-recommends git

# Create the directory .ssh and add github.com to known_hosts
RUN mkdir -p -m 0600 ~/.ssh && \
    ssh-keyscan github.com >> ~/.ssh/known_hosts

# Copy requirements.txt first
COPY requirements.txt .

# Install Python requirements
RUN --mount=type=secret,id=ssh_id_rsa,dst=/root/.ssh/id_rsa pip install -r requirements.txt

# Copy the project
COPY . .

# Run main.py
CMD ["python", "main.py"]

It works perfectly when I build it in this way:

DOCKER_BUILDKIT=1 docker build --secret id=ssh_id_rsa,src=/home/username/.ssh/id_rsa -t my-proj .

But when I try to write docker-compose.yml for it it fails with the error:

#10 1.394   git@github.com: Permission denied (publickey).
#10 1.394   fatal: Could not read from remote repository.

My docker-compose.yml is:

version: '3'

services:
  my-proj:
    build: .
    secrets:
      - ssh_id_rsa
    restart: always

secrets:
  ssh_id_rsa:
    file: /home/username/.ssh/id_rsa

I try to build and run it with the command:

DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose up --build

What is wrong with my docker-compose.yml or with the way I run it?

Docker version 20.10.12, build e91ed57

Docker Compose version v2.2.3