I am having issues creating log files in multiple different docker containers (rabbitmq container, rabbit mongodb BE, and my 2 asynchronous task scheduler services) on my server due to permission errors. I believe it may have something to do with the architecture on which I run the docker containers, reason being, I have no issues when they are brought up on my mac m1 (arm64 ?), but as soon as I attempt to run them on my server (amd64 ?), each container fails to create log files. I will use rabbitmq container, and mongodb container as an example:
docker-compose:
rabbit:
hostname: rabbit
image: "rabbitmq:3-management"
volumes:
- ./data/rabbitmq_logs:/var/log/rabbitmq
env_file:
- .env
ports:
- "15672:15672"
- "5672:5672"
rabbit_mongodb:
image: mongo:latest
volumes:
- ./data/rabbit_mongodb_logs:/var/log/mongodb
command:
- '--logpath'
- '/var/log/mongodb/rabbit_mongodb_log.log'
ports:
- "27017:27017"
.env:
RABBITMQ_LOG_BASE=/var/log/rabbitmq
RABBITMQ_LOGS=${RABBITMQ_LOG_BASE}/rabbit_log.log
RABBITMQ_SASL_LOGS=${RABBITMQ_LOG_BASE}/rabbit_sasl_log.log
mongodb error logs:
rabbit_mongodb_1 | {"t":{"$date":"2021-11-04T16:22:14.797Z"},"s":"F", "c":"CONTROL", "id":20574, "ctx":"-","msg":"Error during global initialization","attr":{"error":{"code":38,"codeName":"FileNotOpen","errmsg":"Can't initialize rotatable log file :: caused by :: Failed to open /var/log/mongodb/rabbit_mongodb_log.log"}}}
cntautomation-private_rabbit_mongodb_1 exited with code 1
rabbitmq error logs:
rabbit_1 | 2021-11-04 16:22:28.227565+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
rabbit_1 | 2021-11-04 16:22:28.327521+00:00 [info] <0.222.0> Feature flags: [ ] implicit_default_bindings
rabbit_1 | 2021-11-04 16:22:28.327573+00:00 [info] <0.222.0> Feature flags: [ ] maintenance_mode_status
rabbit_1 | 2021-11-04 16:22:28.327596+00:00 [info] <0.222.0> Feature flags: [ ] quorum_queue
rabbit_1 | 2021-11-04 16:22:28.327623+00:00 [info] <0.222.0> Feature flags: [ ] stream_queue
rabbit_1 | 2021-11-04 16:22:28.327726+00:00 [info] <0.222.0> Feature flags: [ ] user_limits
rabbit_1 | 2021-11-04 16:22:28.327744+00:00 [info] <0.222.0> Feature flags: [ ] virtual_host_metadata
rabbit_1 | 2021-11-04 16:22:28.327768+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
rabbit_1 | 2021-11-04 16:22:30.826617+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
rabbit_1 | 2021-11-04 16:22:30.826709+00:00 [noti] <0.222.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
rabbit_1 |
rabbit_1 | BOOT FAILED
rabbit_1 | ===========
rabbit_1 | failed to open log file at '/var/log/rabbitmq/rabbit@rabbit_upgrade.log', reason: permission denied
rabbit_1 |
rabbit_1 | 2021-11-04 16:22:31.039651+00:00 [erro] <0.222.0>
rabbit_1 | 2021-11-04 16:22:31.039651+00:00 [erro] <0.222.0> BOOT FAILED
rabbit_1 | 2021-11-04 16:22:31.039651+00:00 [erro] <0.222.0> ===========
rabbit_1 | 2021-11-04 16:22:31.039651+00:00 [erro] <0.222.0> failed to open log file at '/var/log/rabbitmq/rabbit@rabbit_upgrade.log', reason: permission denied
rabbit_1 | 2021-11-04 16:22:31.039651+00:00 [erro] <0.222.0>
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> crasher:
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> initial call: application_master:init/4
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> pid: <0.221.0>
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> registered_name: []
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> exception exit: {{cannot_log_to_file,
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> "/var/log/rabbitmq/rabbit@rabbit_upgrade.log",eacces},
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> {rabbit,start,[normal,[]]}}
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> in function application_master:init/4 (application_master.erl, line 142)
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> ancestors: [<0.220.0>]
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> message_queue_len: 1
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> messages: [{'EXIT',<0.222.0>,normal}]
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> links: [<0.220.0>,<0.44.0>]
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> dictionary: []
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> trap_exit: true
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> status: running
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> heap_size: 610
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> stack_size: 29
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> reductions: 177
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0> neighbours:
rabbit_1 | 2021-11-04 16:22:32.043686+00:00 [erro] <0.221.0>
rabbit_1 | 2021-11-04 16:22:32.067915+00:00 [noti] <0.44.0> Application rabbit exited with reason: {{cannot_log_to_file,"/var/log/rabbitmq/rabbit@rabbit_upgrade.log",eacces},{rabbit,start,[normal,[]]}}
rabbit_1 | {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{{cannot_log_to_file,\"/var/log/rabbitmq/rabbit@rabbit_upgrade.log\",eacces},{rabbit,start,[normal,[]]}}}"}
rabbit_1 | Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{{cannot_log_to_file,"/var/log/rabbitmq/rabbit@rabbit_upgrade.log",eacces},{rabbit,start,[normal,[]]}}})
rabbit_1 |
rabbit_1 | Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...cntautomation-private_rabbit_1 exited with code 0
Again, there are no issues when brought up on my local development server (mac m1) but when trying to bring up on the server (aws lightsail instance), I am having these issues. This is happening with most of my services. I am not super familiar on properly managing permissions, so is there some workaround that I could use, or maybe I am missing something entirely?
Any help would be greatly appreciated. Thank you!
UPDATE:
It is more likely the case that the mounted volume that is getting created is created as root:
drwxr-xr-x 8 root root 4096 Nov 4 22:32 data
and all log directories within the data directory also have permissions set for root.
When entering task scheduling services I see that the logs directory created has permissions:
drwxr-xr-x 2 root root 4096 Nov 4 22:32 logs
but on my mac it has:
drwxr-xr-x 9 nobody nogroup 288 Nov 4 21:40 logs
which is what I set the ownership to in the Dockerfile and the user as for the service in docker-compose file.
worker:
build:
context: ./services/website
dockerfile: Dockerfile.celery
user: nobody
entrypoint: celery -A src.celery.celery:celery worker --loglevel INFO
volumes:
- ./data/worker_logs:/usr/src/app/logs
env_file:
- .env
environment:
- DEPLOYMENT_TYPE=development
- LOG_FILE=./logs/worker_logs.log
depends_on:
- rabbit
- web
- rabbit_mongodb
restart: on-failure:10
FROM python:3.9.7
WORKDIR /usr/src/app
COPY ./requirements.txt .
RUN pip install -r requirements.txt
COPY . .
RUN chown nobody:nogroup ./
RUN ln -s /run/shm /dev/shm
RUN mkdir -p /usr/src/app/logs
RUN chown nobody:nogroup /usr/src/app/logs
It seems like the Docker container is taking the permissions of the mounted volume from the server, but it does not do that on my mac. How am I supposed to set permissions for my docker containers and properly manage these mounted volumes? Note that the default user for aws lightsail instances is "ubuntu".