Goal

I want to build a local docker compose deployment such that I have 5 services.

• Redis
• Postgres
• RabbitMQ
• Django API
• Django Worker

In this deployment the user uploads a file via the API endpoint. This endpoint will store the file in a FileField field in a model.

In a separate transaction the user will trigger an asynchronous task via a separate endpoint. This task will be responsible for

• downloading the file
• extracting the file
• kicking of sub tasks to perform intermediate processing steps
• upload the results of the processing to the database

The intermediate processing steps are NOT supposed to upload any of the files to the database.

The intermediate processing steps are to use django's internal file storage solution to download and upload files there. This is implemented with a file system hierarchy that is not relevant to this question.

The Problem

I have managed to get my local file system working with this configuration. If I run the backend of redis, postgres, and rabbitmq. And then, I run the API and Worker on my machine locally everything runs fine.

When I create a docker-compose configuration and decouple everything. The operation seems to break. And in my docker-compose logging, what I see it:

worker_1  | [2019-10-23 22:27:34,626: WARNING/ForkPoolWorker-2] //--------------------------------------------------------------------------------
worker_1  | [2019-10-23 22:27:34,627: WARNING/ForkPoolWorker-2] // BEGINNING TASK
worker_1  | [2019-10-23 22:27:34,627: WARNING/ForkPoolWorker-2] //--------------------------------------------------------------------------------
worker_1  | [2019-10-23 22:27:34,628: WARNING/ForkPoolWorker-2] // Root Job - 183916ca-f6e6-4e7c-a997-e8f516ccf8be
worker_1  | [2019-10-23 22:27:34,628: WARNING/ForkPoolWorker-2] // Parent Job - None
worker_1  | [2019-10-23 22:27:34,628: WARNING/ForkPoolWorker-2] // Current Job - 183916ca-f6e6-4e7c-a997-e8f516ccf8be
worker_1  | [2019-10-23 22:27:34,628: WARNING/ForkPoolWorker-2] //--------------------------------------------------------------------------------
worker_1  | [2019-10-23 22:27:34,629: WARNING/ForkPoolWorker-2] // PERFORMING DATA SET PRE PROCESSING
worker_1  | [2019-10-23 22:27:34,629: WARNING/ForkPoolWorker-2] //--------------------------------------------------------------------------------
worker_1  | [2019-10-23 22:27:34,629: WARNING/ForkPoolWorker-2] {'data_set_id': 1, 'starting_node': 'Live', 'organization_id': 1}
worker_1  | [2019-10-23 22:27:34,630: WARNING/ForkPoolWorker-2] Downloading the files required to run!
worker_1  | [2019-10-23 22:27:34,645: WARNING/ForkPoolWorker-2] Downloading remote file `organizations/1/data_sets/flow_cytometry/triple_hello_world_payload.tgz`
worker_1  | [2019-10-23 22:27:34,646: WARNING/ForkPoolWorker-2] Exists: `False`
worker_1  | [2019-10-23 22:27:34,646: WARNING/ForkPoolWorker-2] ERROR occured: [Errno 2] No such file or directory: '/opt/api_webserver/media/organizations/1/data_sets/flow_cytometry/triple_hello_world_payload.tgz'.
worker_1  | [2019-10-23 22:27:34,653: INFO/ForkPoolWorker-2] Task api.versions.v1.tasks.main_task.main_task[183916ca-f6e6-4e7c-a997-e8f516ccf8be] succeeded in 0.02647909999359399s: {'iteration': 0, 'completion': 0, 'status': 'ERROR', 'message': 'Excecuting `main_task` failed!', 'error': 'Error in `main_task`: [Errno 2] No such file or directory: \'/opt/api_webserver/media/organizations/1/data_sets/flow_cytometry/triple_hello_world_payload.tgz\'.'}

If I go into the worker docker container and check the file system the path does NOT exist for the media directory and file.
If I go into the api docker container and check the file system the path does exist for the media directory and file.

Relevant Code

I won't be supplying view code, or api code, as the API works fine.

Upload and file retrieval is handled in the workers process by using django's default_storage interface.

Django's Default Storage Interface API

The issue is related to the worker, so here is some of the related code.

worker.py

# Python Standard Libraries
import os
# Third-Party Libraries
import tempfile
# Custom
from models.data_set_model import DataSet
from tasks.helpers import download_remote_file


def download_data_set(data_set_id):
    print("Downloading the files required to run!")
    data_set = DataSet.objects.get(id=data_set_id)

    remote_file_path = data_set.file.name
    remote_file_name = os.path.basename(remote_file_path)

    temporary_directory_path = tempfile.mkdtemp()
    temporary_compressed_file_path = os.path.join(temporary_directory_path, remote_file_name)

    download_remote_file(remote_file_path, temporary_compressed_file_path)

    return temporary_compressed_file_path

helpers.py

# Python Standard Libraries
# N/A
# Third-Party Libraries
from django.core.files.storage import default_storage
# CustomLibraries
# N/A


def download_remote_file(remote_file_path, local_file_path):
    print(f"Downloading remote file `{remote_file_path}`")
    print(f"Exists: `{default_storage.exists(remote_file_path)}`")
    remote_file_contents = None
    with default_storage.open(remote_file_path) as remote_file_handle:
        print("Reading file contents")
        remote_file_contents = remote_file_handle.read()

    print(f"Placing remote file contents into `{local_file_path}`")
    with open(local_file_path, "wb") as local_file_handle:
        local_file_handle.write(remote_file_contents)

Outstanding Questions

• What am I doing wrong?
• What is the idiomatic way to download files on the worker from the API's FileStorage system?
• Shouldn't the default_storage.open() command point to the API's file system and be able to download it
  • If it isn't are there configurations I can make on the worker to support this?
• Does this only work locally because the file system is shared, and the reason this is happening is because docker-compose is breaking these into separate environments?