Traefik load balance between Docker provider and server at url

I currently have a Traefik setup with one nodejs service running locally in a docker container with a docker-compose.yml file like so:

container_name: nestjs-server
build:
  context: ./
  dockerfile: Dockerfile
networks:
  - traefik-global-proxy
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.nestjs-server.rule=Host(`mydomain.com`) || Host(`www.mydomain.com`)"
  - "traefik.http.routers.nestjs-server.entrypoints=websecure"
  - "traefik.http.routers.nestjs-server.tls.certresolver=letsencrypt"
  - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

I am running on a 1vCPU / 2GB cloud instance. Now I would like to add a second node app instance on another VM. I have seen it is possible to add server instances to the load balancer like so:

  services:
    my-service:
      loadBalancer:
        servers:
        - url: "http://<private-ip-server-1>:<private-port-server-1>/"
        - url: "http://<private-ip-server-2>:<private-port-server-2>/"

But I am not sure how to load balance between instances on another service alongside local docker instances. I have read that it is not possible to mix the label based config with the file config, so I assume I'd need to do something like:

    my-service:
      loadBalancer:
        servers:
        - url: "http://<private-ip-server-1>:<private-port-server-1>/"
        - port: "<local-port-server-2>"

Is this possible? What is the correct way to accomplish this, other than having everything run on the same machine within docker?