Docker (v20.10.17, install following official docs) port forwarding stops working after stopping container and restarting it? (Requires full reinstall to get working again).
Creating postgres docker container in Ubuntu 18.04 like...
docker run -it \
-e POSTGRES_USER="root" \
-e POSTGRES_PASSWORD="root" \
-e POSTGRES_DB="ny_taxi" \
-v /my/absolute/path/to/my/local/folder/ny_taxi_postgres_data:/var/lib/postgresql/data:rw \
-p 5432:5432 \
postgres:13
...but can't connect when using psql
➜ ~ psql -h localhost -p 5432 -U root -W
Password for user root:
psql: connection to server at "localhost" (127.0.0.1), port 5432 failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
or using pgadmin4 from my local desktop or changing "localhost" to the value "0.0.0.0"
I find that after doing a fresh install of docker following the info here ( https://askubuntu.com/a/1021506/760862 ) then re-running the sudo apt-get install command for the docker packages,...
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
...I was able to psql into the container's DB, but only for the first time of running that container. Eg. after ctl+C'ing the container, re-running the exact same docker run command, I could no longer psql into the DB (getting the same "psql: connection to server at "localhost" (127.0.0.1), port 5432 failed: server closed the connection unexpectedly" error I've been seeing).
My linux docker group info looks like:
➜ cat /etc/group | grep docker
docker:x:999:myusername
I see no local postgres service running:
➜ ~ systemctl status postgresql
Unit postgresql.service could not be found.
➜ ~ service postgresql status
Unit postgresql.service could not be found.
and this is the only thing that appears to be using the specified port:
➜ sudo lsof -i -P -n | grep LISTEN
.
.
.
docker-pr 5125 root 4u IPv4 3785093 0t0 TCP *:5432 (LISTEN)
docker-pr 5132 root 4u IPv6 3778439 0t0 TCP *:5432 (LISTEN)
➜
➜
➜
➜ sudo netstat -tulpen | grep 5432
tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 0 4897282 15384/docker-proxy
tcp6 0 0 :::5432 :::* LISTEN 0 4897287 15391/docker-proxy
This is also the only docker container I have running and the image was just downloaded today:
➜ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
033699b59c10 postgres:13 "docker-entrypoint.s…" 3 hours ago Up 3 hours 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp lucid_mcclintock
The logged output I see from the docker container is
➜ docker container logs lucid_mcclintock
PostgreSQL Database directory appears to contain a database; Skipping initialization
2022-08-16 20:51:39.219 UTC [1] LOG: starting PostgreSQL 13.8 (Debian 13.8-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
2022-08-16 20:51:39.219 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-08-16 20:51:39.219 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-08-16 20:51:39.231 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-08-16 20:51:39.249 UTC [26] LOG: database system was shut down at 2022-08-16 20:51:34 UTC
2022-08-16 20:51:39.261 UTC [1] LOG: database system is ready to accept connections
(I'm pretty sure the "database system was shut down at" log message is just from the fact that I ctl+Ced out of the container the previously).
This log output for the DB container does not change even when trying to psql connect using a wrong username or password (eg. having some message about a failed login attempt) -- thus making me suspect some kind of port forwarding issue with docker.
IDK how much this helps (b/c not much experience with docker), but the only container I see on the docker Bridge network (via docker network inspect bridge) is the container name of the one running the DB (ie. no hidden container also using the network).
I also see:
➜ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
033699b59c10 postgres:13 "docker-entrypoint.s…" 4 hours ago Up 4 hours 0.0.0.0:5432->5432/tcp, :::5432->5432/tcp lucid_mcclintock
➜ docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' 033699b59c10
172.17.0.2
This issue appears to affect all port forwarding with docker containers on my computer. Eg. I also have issues now doing a test like like
➜ docker run -p 9999:80 httpd
While in a separate terminal running:
➜ curl localhost:9999
curl: (56) Recv failure: Connection reset by peer
My docker version is
➜ docker version
Client: Docker Engine - Community
Version: 20.10.17
API version: 1.41
Go version: go1.17.11
Git commit: 100c701
Built: Mon Jun 6 23:02:56 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Checking the IPTables output (in response to comments (I don't really understand any implications of the output)), I see:
➜ sudo iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere 172.17.0.2 tcp dpt:postgresql
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
This seems really weird, but I have little experience with docker or networking. Does anyone have any debugging suggestions, more diagnostics that should be added here, or know what could be going wrong?