Host key verification failed when building image with docker-compose from a private repo over ssh

Inside my docker-compose.yml I'm using ssh address context to build a Docker image.

ui:
    build:
      context: git@my.git.host:ui/ui.git
      dockerfile: Dockerfile.prod
    container_name: ui
    ports:
      - "4200:4200"
    command: ember serve

Running docker-compose build ui gives me :

compose.cli.verbose_proxy.proxy_callable: docker build -> <generator object APIClient._stream_helper at 0x7faa8bc107d8>
ERROR: compose.cli.errors.log_api_error: error fetching: Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

If I build the image using docker build it works normally

docker build git@my.git.host:ui/ui.git -f Dockerfile.prod

My context

compose.cli.command.get_client: docker-compose version 1.21.0, build unknown
docker-py version: 4.0.1
CPython version: 3.6.7
OpenSSL version: OpenSSL 1.1.1  11 Sep 2018

LATER EDIT:
I managed to reproduce the issue using the docker pip

import docker
client = docker.APIClient()
log = client.build(path="git@my.git.host:ui/ui.git", dockerfile="Dockerfile.prod")
for chunk in log:
  print(loc)

Output:

docker.errors.APIError: 500 Server Error: Internal Server Error ("error fetching: Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
: exit status 128")