Postgres with docker loses connection and gets filesystem issue

I am using Postgres 9.4.1 with a Docker 17.06.0 and docker-compose 1.14.0. When I work with it, it often happens to just lose connection and in logs I get this:

LOG:  invalid length of startup packet
LOG:  could not send data to client: Broken pipe
FATAL:  connection to client lost
ERROR:  could not open file "base/1/11943": No such file or directory
FATAL:  could not open file "base/12141/11943": No such file or directory

After restarting container it doesn't get any better:

postgres cannot access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory

My only solution is to:

1. Stop container and remove it
2. Remove all data of volumes
3. Restart docker
4. Up container.

To be honest it's quite annoying process it. It happens always when I disconnect with my current network and sometimes just when I work with postgres. Maybe I'm missing some permission configuration, this is my docker-compose.yml:

postgres:
 image: postgres:9.4.1
 ports:
   - 54320:54320
 environment:
   - POSTGRES_USER=xxx
   - POSTGRES_PASSWORD=xxx
 volumes:
   - /tmp/postgres/staging:/var/lib/postgresql/data
 restart: always