Django: Uploaded files not being garbage collected, causing memory problems

Viewed 859

I have a Django view which handles uploaded files, and when it is called repeatedly, we always encounter over memory errors. (We're on Heroku, so we get 512mb of memory per web dyno).

The over memory errors show that each call increases memory usage by the size of the file, and memory usage never goes back down. I think the server reads the image into memory to save it into a Django model, but then never releases the memory when it's done.

I think there must be a reference to the image that is persisting after the function returns, preventing the image from being garbage collected. But I can't think of what that could be.

Here's a simplified version of the code, which I tested to make sure it has the same problem:

def simplified_add_image(request, biz_id):
    if request.is_ajax():
        # the file is stored raw in the request
        newBI = NewBusinessImage(business_id=biz_id, name=request.GET.get("name"), primary_image=True)
        newBI.id = uuid.uuid4()
        newBI.save()
        uniquename = biz_id + ".." + get_a_uuid() + ".jpg"

        newBI.original_image.save(uniquename, ContentFile(request.read()))

        # this starts a series of tasks to process the image into various sizes.
        # don't think it's the problem because it runs on a separate server, and the
        # web server is the one that goes over memory
        tasks.image_meta_task.delay(uniquename, newBI.id)

        return StockJSONResponse(request, {
            "success" : True,
        })

I'd really appreciate any help. Thanks a lot!

Clay

The following is additional information that was requested in comments:

We are storing our files on AmazonS3:

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
STATIC_URL = 'https://s3.amazonaws.com/%s/' % AWS_STORAGE_BUCKET_NAME
from S3 import CallingFormat
AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN

And here is the NewBusinessImage model

class NewBusinessImage(models.Model):
    id = UUIDField(auto=True,primary_key=True,version=4, default=uuid.uuid4())
    business = models.ForeignKey('Business')
    name = models.CharField(max_length=100)
    original_image = models.ImageField(upload_to='photos/originals/')
    thumbnail = models.ImageField(upload_to='photos/thumbnails/')
    display_image = models.ImageField(upload_to='photos/displays/')
    enlarged_image = models.ImageField(upload_to='photos/enlarged/')
    num_views = models.PositiveIntegerField(editable=False, default=0)
    primary_image = models.BooleanField(default=False)
1 Answers
Related