How to release memory resources after django-background-task completes its execution?

user_5919632 0
Viewed 336

I'm executing some of the long-running tasks with Django-background-tasks. Background task takes some data from DB and process it internally which requires memory of 1 GB for each task.

The task is running in the async way.

When any task runs and completes its execution, Django-background-tasks does not release the memory after completing execution. Which causing memory issues to the Dameon pod. Any solution around deallocation of memory resources in Django-background-task? E.g.

Note: resources available are fixed. so need a workaround on resource deallocation of tasks. Currently, memory gets deallocated when the pod restarts.

This is how module structure looks like.

../bulk_file_operations/
├── apps.py
├── tasks.py
├── views.py
├── file_handler.py

tasks.py

@background()
def my_task():
    """
    a long-running task that takes time of 20-30 mins, which includes some DB interactions.
    """
    process_data_n_download(model_queryset_object)
    pass

file_handler.py


def process_data_n_download(queryset_object):
    """process this ORM object for file creation."""
    pass
multithreading django python
1 Answers

You can't explicitly free memory. What you need to do is to make sure you don't keep references to objects. They will then be garbage collected, freeing the memory.

In your case,

  1. if you need large lists, you typically need to reorganize the code, typically using generators/iterators instead. That way you don't need to have the large lists in memory at all. http://www.prasannatech.net/2009/07/introduction-python-generators.html

  2. Check if django-background-tasks app's internals if it is keeping any references. also, check your code too. You can use the del keyword to remove the references if any.

user_92752490
Related