Share media between multiple django(VMs) servers

user_7730705 0
Viewed 184

We have deployed a django server (nginx/gunicorn/django) but to scale the server there are multiple instances of same django application running.

Here is the diagram (architecture):

enter image description here

Each blue rectangle is a Virtual Machine.

HAProxy sends all request to example.com/admin to Server 3.other requests are divided between Server 1 and Server 2.(load balance).

Old Problem:

Each machine has a media folder and when admin Uploads something the uploaded media is only on Server 3. (normal users can't upload anything)

We solved this by sending all requests to example.com/media/* to Server 3 and nginx from Server3 serves all static files and media.

Problem right now

We are also using sorl-thumbnail.

When a requests comes for example.com/,sorl-thumbnail tries to access the media file but it doesn't exist on this machine because it's on Server3.

So now all requests to that machine(server 1 or 2) get 404 for that media file.

One solution that comes to mind is to make a shared partition between all 3 machines and use it as media. Another solution is to sync all media folders after each upload but this solution has problem and that is we have almost 2000 requests per second and sometimes sync might not be fast enough and sorl-thumbnail creates the database record of empty file and 404 happens.

Thanks in advance and sorry for long question.

django python
2 Answers

You should use an object store to save and serve your user uploaded files. django-storages makes the implementation really simple.

If you don’t want to use cloud based AWS S3 or equivalent, you can host your own on-prem S3 compatible object store with minio.

On your current setup I don’t see any easy way to fix where the number of vm s are dynamic depending on load.

If you have deployment automation then maybe try out rsync so that the vm takes care of syncing files with other vms.

user_3316550

Question: What was the problem?

we got 404 on other machines because normal requests (requests asking for a template) would get a 404 not found on thumbnail media. real problem was with sorl-thumbnail template tags.

Here is what we ended up doing:

In models that needed a thumbnail, we added functions to create that specific thumbnail.

and using a post-save signal in the admin machine called all those functions to make sure all the thumbnails were created after save and the table for sorl-thumbnail is filled.

now in templates instead of calling sorl-thumbnail template tags now we call a function in model.

user_77307050
Related