Copy data from host to container efficiently

Nero_f93 0
Viewed 22

The idea is a docker container which aims to train machine learning models for computer vision. The data which is trained needs to be uploaded to the container, consumed and deleted afterwards.

Is there any way to use a volume and transport data efficiently between the host and the container?

When I searched on the web, most sources mentioned manual transport via bash or something similar, while my application needs to do this in an automated an repeating way for different datasets.

The host machine is a windows and the container is linux.

EDIT: there is a main application running on the machine which is responsible for managing the process.

  • send data to container (somehow?)
  • trigger training process

the training process runs async to not block the rest api

Any ideas?

containers docker
1 Answers

You can use bind mounts when building you docker container, See https://docs.docker.com/storage/bind-mounts/

you specify a directory inside your container which will be mounted on a local directory on your host, this way all container write operations are mounted directly to your host directory.

I'd recommand doint it using docker-compose, for example below:

container directory /opt/Projects/01_MyProject will be mounted to host directory /Volumes/Partition_2/Docker-Volumes/Volume_1,

#docker-compose.yml
...
    volumes:
      - PERSIST_DATA:/opt/Projects/01_MyProject
volumes:
  PERSIST_DATA:
    driver: local
    driver_opts:
      o: bind
      type: none
      device: /Volumes/Partition_2/Docker-Volumes/Volume_1

This way, you can copy anything to /Volumes/Partition_2/Docker-Volumes/Volume_1, which will show up under /opt/Projects/01_MyProject

Reda E.0
Related