Docker push is failing when I have large HDF5 files in the image. One file is between 2.4 and 3.2 GB. It doesn't matter which file it is, it will always fail. Small HDF5 files (19MB) will succeed.
I actually do not get any error message if I attempt to push to the docker registry. The UI looks like it is pushing the file, then when it has pushed all the bytes (100%), it says "Retrying in 15s", and then starts all over again.
I have some error logs from the gitlab-registry below, but a "docker push --verbose" method would be quite helpful.
==> /var/log/gitlab/registry/current <== 2017-08-09_09:23:22.28799 time="2017-08-09T09:23:22.287891292Z" level=error msg="client disconnected during blob PATCH" auth.user.name=root contentLength=2000110499 copied=1432460362 environment=production error="unexpected EOF" go.version=go1.8.1 http.request.host=registry.gitlab.bignut.ai http.request.id=cb09cab1-8a4c-4001-9aa4-7c11dc9c04c9 http.request.method=PATCH http.request.remoteaddr=35.189.251.28 http.request.uri="/v2/root/gitlab-docker-test/wa_tr/wa_tr_test_data/blobs/uploads/d463eda2-4d84-4b0a-a76a-37dfc043d750?_state=ZTIcKZ18tNeK8HAEIyt4iy0zpqDTC3L6h7phBly07S97Ik5hbWUiOiJyb290L2dpdGxhYi1kb2NrZXItdGVzdC93YV90cmFkaW5nL3dhX3RyYWRpbmdfdGVzdF9kYXRhIiwiVVVJRCI6ImQ0NjNlZGEyLTRkODQtNGIwYS1hNzZhLTM3ZGZjMDQzZDc1MCIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAxNy0wOC0wOVQwOToxOTowNC43ODE1NzIwMTlaIn0%3D" http.request.useragent="docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/4.4.0-89-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce (linux))" instance.id=070c4d24-6572-472d-81a4-0ecccd173789 service=registry vars.name="root/gitlab-docker-test/wa_tr/wa_tr_test_data" vars.uuid=d463eda2-4d84-4b0a-a76a-37dfc043d750 version=v2.6.1-1-gdd544a8 2017-08-09_09:23:23.05762 127.0.0.1 - - [09/Aug/2017:09:22:42 +0000] "PATCH /v2/root/gitlab-docker-test/wa_tr/wa_tr_test_data/blobs/uploads/d463eda2-4d84-4b0a-a76a-37dfc043d750?_state=ZTIcKZ18tNeK8HAEIyt4iy0zpqDTC3L6h7phBly07S97Ik5hbWUiOiJyb290L2dpdGxhYi1kb2NrZXItdGVzdC93YV90cmTM3ZGZjMDRhcnRlZEF0IjoiMjAxOToxOTowNC43ODE1NzIwMTlaIn0%3D HTTP/1.0" 499 0 "" "docker/17.06.0-ce go/go1.8.3 git-commit/02c1d87 kernel/4.4.0-89-generic os/linux arch/amd64 UpstreamClient(Docker-Client/17.06.0-ce (linux))"
Output of docker version:
Client: Version: 17.06.0-ce API version: 1.30 Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:23:31 2017 OS/Arch: linux/amd64
Server: Version: 17.06.0-ce API version: 1.30 (minimum version 1.12) Go version: go1.8.3 Git commit: 02c1d87 Built: Fri Jun 23 21:19:04 2017 OS/Arch: linux/amd64 Experimental: false
Containers: 0 Running: 0 Paused: 0 Stopped: 0 Images: 28 Server Version: 17.06.0-ce Storage Driver: aufs Root Dir: /var/lib/docker/aufs Backing Filesystem: extfs Dirs: 40 Dirperm1 Supported: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: cfb82a876ecc11b5ca0977d1733adbe58599088a runc version: 2d41c047c83e09a6d61d464906feb2a2f3c52aa4 init version: 949e6fa Security Options: apparmor seccomp Profile: default Kernel Version: 4.4.0-89-generic Operating System: Ubuntu 16.04.2 LTS OSType: linux Architecture: x86_64 CPUs: 16 Total Memory: 91.47GiB Name: amitt ID: 5PGY:FXXK:WB4M:YBZP:SQZU:DIZD:6WYC:CZ6T:NZZ5:3C7D:WQHX:O3UB Docker Root Dir: /var/lib/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false
WARNING: No swap limit support