Can I use docker-compose without a bash script to inspect a container then copy files to host from the container?

user_4005959 0
Viewed 76

This is the workflow I'm trying to parallelize:

  1. docker build
  2. docker run
  3. docker inspect container -> returns false
  4. docker cp container:file /host/

I was wanting to use docker-compose to do this on a single host, then transition to Kubernetes later so that I can orchestrate this on multiple hosts.

Should I create a bash script and have it RUN in the Dockerfile?

I'm looking for a solution that the community accepts as the best practice.

dockerfile docker-swarm docker-compose kubernetes docker
1 Answers

In single-host-Docker land, you can try to arrange things so that docker run does everything you need it to. Avoid docker inspect (which dumps out low-level diagnostics that usually aren't interesting) and docker cp. Depending on how you build things, you could build the artifact in your Dockerfile, and copy it out

docker build -t the-image .
# use "cat" to get it out?
docker run --rm the-image \
  cat /app/file \            # from the container
  > file                     # via the container's stdout, to the host
# use volumes to get it out?
docker run --rm -v $PWD:/host the-image \
  cp /app/file /host

Depending on what you're building, you might extend this further to pass both the inputs and outputs in volumes, so the image is just a toolchain. For a minimal Go application, using the Docker Hub golang image, for example:

# don't docker build anything, but run a container with the toolchain
docker run --rm \
  -v $PWD:/app \
  -w /app \
  golang:1.15 \
  go build -o the_app ./cmd/the_app

In this last setup the -w working directory is the bind-mounted /app directory, so go build -o ./the_app writes out to the host.

Since this setup is a little more oriented towards single-use containers (note the docker run --rm option) it's not a good match for Compose, which generally expects long-running server-type containers.

This setup also will not translate well to Kubernetes. Kubernetes isn't really good at sharing files between containers or with systems outside the cluster. If you happen to have an NFS server you can use that, but there aren't native options; most of the volume types that it's straightforward to get are ReadWriteOnce volumes that can't be reused between multiple Kubernetes Pods (containers).

You could in principle write a Kubernetes Job that did a single compilation. It can't run docker build, so the "run" step would have to do the actual building. You can't kubectl cp out of a completed pod (see e.g. kubernetes/kubectl#454), so it needs to send its content somewhere specific when it's done.

A better high-level approach here would be to find or install some sort of network-accessible storage, especially to hold the results (an Artifactory server; object storage like AWS S3). Rewrite your build sequence as a "task" that takes the locations of the inputs and outputs and runs the build, ignoring the local filesystem. Set up a job queue like RabbitMQ, and inject the tasks into the queue. Finally, run the builder-worker as a Kubernetes Deployment; it will build as many things in parallel as the number of replicas: in the deployment.

user_100081730
Related