TLDR: How can one build docker images with Gradle inside a Openjdk container?
The problem
We are building a Micronaut based suite of microservices in a multi-module Gradle project using GitLab pipelines.
Currently our pipeline builds docker images in a separate pipeline stage using docker command line, but it is starting to become unwieldy and cumbersome to add new services that way.
So instead of having separate pipeline steps for building the executable and then building docker images for each executable, I am looking into building the docker images right along with the main build step using gradle.
Micronaut's gradle plugin includes and extends gradle-docker-plugin and allows building docker images using dockerBuild task.
The whole Gradle build step is executed from within openjdk:14 docker image by custom private gradle-runner instance, so that container doesn't have any docker related bits on it.
What I've tried so far
The naive approach
My first attempt was to simply add dockerBuild target to the Gradle command line. That one failed as expected with the following stacktrace (abbreviated for the sake of clarity):
Execution failed for task ':my-service:dockerBuild'.
> com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
* Try:
Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':my-service:dockerBuild'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:188)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
/.../
at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51)
> Task :my-service:dockerBuild FAILED
Building image using context '/builds/my-group/my-project/my-service/build/docker'.
Using Dockerfile '/builds/my-group/my-project/my-service/build/docker/Dockerfile'
Using images 'registry.gitlab.com/my-group/my-project/my-service:1.0.1-SNAPSHOT+4b9f8460.179'.
at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57)
/.../
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: java.lang.RuntimeException: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:153)
at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
Caused by: com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.HttpHostConnectException: Connect to http://127.0.0.1:2375 [/127.0.0.1] failed: Connection refused
at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory$1.run(PlainConnectionSocketFactory.java:87)
at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:84)
/.../
at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67)
at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:149)
... 3 more
After some initial headschratching I figured out that this was basically caused by the fact that a build running in a container simply did not have access to docker engine remote api.
Docker socket approach
Couple of mediocre applications of Google-Fu later, I found the recommendation to map /var/run/docker.sock into the container file system.
This prompted me to modify my gitlab-runner configuration:
[[runners]]
name = "gennet-elab jdk runner"
url = "https://gitlab.com/"
executor = "docker"
[runners.docker]
image = "openjdk:14.0.2-slim"
volumes = ["/var/run/docker.sock:/var/run/docker.sock", ...]
Running the pipeline with that configuration yielded me the following failure stacktrace:
Execution failed for task ':my-service:dockerBuild'.
> java.io.IOException: native write() failed : Connection reset by peer
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':my-service:dockerBuild'.
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:188)
at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282)
at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:186)
[...]
at org.gradle.internal.concurrent.ThreadFactoryImpl$ManagedThreadRunnable.run(ThreadFactoryImpl.java:61)
Caused by: java.lang.RuntimeException: java.io.IOException: native write() failed : Connection reset by peer
at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:153)
at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
Caused by: java.io.IOException: native write() failed : Connection reset by peer
at com.github.dockerjava.httpclient5.UnixDomainSocket$UnixSocketOutputStream.write(UnixDomainSocket.java:319)
at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
[...]
at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67)
at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:149)
... 3 more
Suppressed: java.io.IOException: native write() failed : Broken pipe
at com.github.dockerjava.httpclient5.UnixDomainSocket$UnixSocketOutputStream.write(UnixDomainSocket.java:319)
at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117)
at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126)
[...]
at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:8)
at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
at java.base/java.lang.Thread.run(Thread.java:832)
This suggests to me, it tries to connect over the socket, but the connection is closed by docker API server. From reading the documentation it seems there is some sort of
From all that I can understand, the problem might have something to do with docker api security, but all of the tutorials out there show dind (Docker in Docker) usage and that does not really help my use case.
I've also tried running the container with privileged access and tried once with dind service attached to the build job, but none of these seemed to work.
The Question
So, the question for the Great and Powerful Internet is if there is a way to build docker images using Gradle inside openjdk container and leveraging Docker remote API?
If yes, what am I missing? If no, what are the alternatives (yes, I know I can probably just create a separate gitlab shell runner, but I'd like to explore the Docker option first)