I am running a Django application in Heroku which handles multiple image uploads from the user and storage them to Amazon S3. The problem is that the execution of this process requires usually more than 30s (the time execution limit of Heroku).
I tested it and the line which takes longer time is the one saving the image file in the ImageField. This is done like that because the image has to be cropped and processed by ProcessImageFile(). However this function is not taking long time, but the save method itself, maybe because it stores the files in S3 one by one while saving them.
Here is the view (omitted irrelevant lines):
@login_required
def image_create(request):
if request.method == 'POST':
images = request.FILES.getlist("files")
crop_points = json.loads( request.POST.get('crop_points'))
#Validation of inputs in the form: images and other fields
if len(images) < 3 : return JsonResponse({'val_result': 'min_error'})
if len(images) > 12: return JsonResponse({'val_result': 'max_error'})
#We Create the gallery, iterate over the images provided by the form, validate, insert custom fields and save them in bulk associating it to the gallery.
with transaction.atomic():
new_items = []
gallery = Gallery.objects.create( user=request.user )
for i, img_file in enumerate(images):
new_item = Image()
new_item.user = request.user
#-----THIS IS THE PART WHICH TAKES MOST OF THE VIEW PROCESSING TIME: IT IS NOT THE ProcessImageFile FUNCTION, BUT THE SAVE METHOD ITSELF
new_item.image.save( 'img'+ str(i) + '.jpg', content = ProcessImageFile(img_file, crop_points), save=False )
#---------------------------------------------------------------------------------------------------------------------------------------
new_items.append( new_item )
created_objects = Image.objects.bulk_create( new_items )
Belonging.objects.bulk_create( [ Belonging(gallery=gallery, content_id = item.id) for item in new_items] )
for img in created_objects:
img.create_tags(gallery = gallery) #<-We save the notifications for bulk create
return JsonResponse({'status': 'ok', 'gallery': gallery.id})
else:
form = MultiUploadImageForm()
return render(
request,
'upload/create.html',
{'form': form}
)
#I THOUGHT THIS COULD BE THE FUNCTION TAKING TIME BUT IT IS NOT:
def ProcessImageFile(img_file, crop_points):
img = ImageProcessor.open(img_file)
cropped_img = img.crop( ( int(crop_points[0]), int(crop_points[1]), int(crop_points[2]), int(crop_points[3])))
img_io = BytesIO()
cropped_img.save( img_io, format='JPEG', quality=100)
return ContentFile( img_io.getvalue())
I already tried to use Celery for handling the file upload in a separate task, but the problem here is passing the request or image files to the task, since they has to be serialized. Anyway I guess there is something inefficient here and that this simple view shouldn´t be taking more than 30sec for uploading five images in S3 and giving back a response. Maybe the solution would be sending all the images together to S3 in a bulk, or saving them in other way, I don't know.