Django-Compressor won't work in offline mode with custom S3 domain

user_287893 0
Viewed 273

I get the following error whenever a custom domain for the S3 endpoint is used.

#  WORKS 
AWS_S3_CUSTOM_DOMAIN = 'example.fra1.digitaloceanspaces.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
#  DOES NOT WORK ⁉️ 
AWS_S3_CUSTOM_DOMAIN = 'cdn.example.com/{}'.format(AWS_STORAGE_BUCKET_NAME)

CommandError: An error occurred during rendering /home/<user>/<app>/templates/public/index.html: 'https://cdn.example.com/storage/static/node_modules/nouislider/distribute/nouislider.min.css' isn't accessible via COMPRESS_URL ('https://example.fra1.digitaloceanspaces.com/storage/static/') and can't be compressed

If I go to either url the file is accessible, hence its likely that the CDN is ok, URLs are correctly defined, CORS are fine too. Also without django-compressor subdomain delivery had been working fine, leading me to believe the issue is not with django-storages

I've been trying for several hours and ultimately had to do a temporary fix by setting the AWS_S3_CUSTOM_DOMAIN to be the same as the AWS_S3_ENDPOINT_URL. However this is not ideal.

Please see the implementation below.

/requirements.txt

Django==3.1.4
...
boto3~=1.16.46
botocore~=1.19.46
s3transfer~=0.3.3
...
django-storages~=1.11.1
django-compressor~=2.4

/config/settings.py

...
# ==========================================================
#  DJANGO-STORAGES
# ==========================================================
if LOCALHOST_MODE:
    MEDIA_URL = '/media/'
    MEDIA_ROOT = os.path.join(BASE_DIR, 'media')

    STATIC_URL = '/static/'
    STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static/'), ]
else:
    AWS_ACCESS_KEY_ID = os.getenv("AWS_ACCESS_KEY_ID")
    AWS_SECRET_ACCESS_KEY = os.getenv("AWS_SECRET_ACCESS_KEY")

    AWS_STORAGE_BUCKET_NAME = 'storage'
    AWS_S3_ENDPOINT_URL = 'https://example.fra1.digitaloceanspaces.com'
   
    #  WORKS ⚠️ 
    AWS_S3_CUSTOM_DOMAIN = 'example.fra1.digitaloceanspaces.com/{}'.format(AWS_STORAGE_BUCKET_NAME)
    #  DOES NOT WORK ⁉️ 
    AWS_S3_CUSTOM_DOMAIN = 'cdn.example.com/{}'.format(AWS_STORAGE_BUCKET_NAME)  

    AWS_S3_OBJECT_PARAMETERS = {
        'CacheControl': 'max-age=86400',
    }
    AWS_LOCATION = 'static'
    AWS_DEFAULT_ACL = 'public-read'

    STATICFILES_DIRS = (os.path.join(BASE_DIR, "static"), )
    STATIC_URL = '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, AWS_LOCATION)
    STATICFILES_STORAGE = 'storage_backends.CachedS3Boto3Storage'

    MEDIA_ROOT = os.path.join(BASE_DIR, "media")
    MEDIA_URL = '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, 'media')
    DEFAULT_FILE_STORAGE = 'storage_backends.MediaStorage'

# ==========================================================
# DJANGO-COMPRESSOR
# ==========================================================
COMPRESS_ENABLED = True
STATIC_DEPS = True
COMPRESS_ROOT = os.path.join(BASE_DIR, "static")

COMPRESS_FILTERS = {
    'css': ['compressor.filters.css_default.CssAbsoluteFilter', 'compressor.filters.cssmin.rCSSMinFilter'],
    'js': ['compressor.filters.jsmin.JSMinFilter']
}

if LOCALHOST_MODE:
    COMPRESS_OFFLINE = False
else:
    COMPRESS_OFFLINE = True
    COMPRESS_STORAGE = STATICFILES_STORAGE
    COMPRESS_URL = STATIC_URL

STATICFILES_FINDERS = [
    'django.contrib.staticfiles.finders.FileSystemFinder',
    'django.contrib.staticfiles.finders.AppDirectoriesFinder',
    'compressor.finders.CompressorFinder',
]
...

/storage_backends.py

from django.core.files.storage import get_storage_class
from storages.backends.s3boto3 import S3Boto3Storage

from config import settings


class MediaStorage(S3Boto3Storage):
    bucket_name = settings.AWS_STORAGE_BUCKET_NAME
    location = 'media'


class CachedS3Boto3Storage(S3Boto3Storage):
    def __init__(self, *args, **kwargs):
        super(CachedS3Boto3Storage, self).__init__(*args, **kwargs)
        self.local_storage = get_storage_class(
            "compressor.storage.CompressorFileStorage")()

    def save(self, name, content):
        self.local_storage._save(name, content)
        super(CachedS3Boto3Storage, self).save(name, self.local_storage._open(name))
        return name
django-compressor django-storage boto cdn django
2 Answers

How to debug this error:

Looking at where the error is raised (in django-compressor.compressor.base) we find the following:

    def get_basename(self, url):
        try:
            base_url = self.storage.base_url
        except AttributeError:
            base_url = settings.COMPRESS_URL

        base_url = str(base_url)
        if not url.startswith(base_url):
            raise UncompressableFileError(
                "'%s' isn't accessible via "
                "COMPRESS_URL ('%s') and can't be "
                "compressed" % (url, base_url)
       )

Looking at your settings:

  • your COMPRESS_URL is set equal to STATIC_URL,
  • which in turn is equal to '{}/{}/{}/'.format(AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME, AWS_LOCATION).
  • This is https://example.fra1.digitaloceanspaces.com/storage/static/
  • But the url being passed in is https://cdn.example.com/storage/static/...

This is the route of the problem.

So what's happening??

  • COMPRESS_URL controls the URL that linked files will be read from and compressed files will be written to.
  • When you have a CSS or JS file with a link, rel or src attribute somewhere, django-compressor calls get_basename.
  • get_basename takes full path to a static file (eg. "/static/css/style.css") and returns path with storage's base url removed (eg. "css/style.css").
  • Since the base path of some link, rel or src attribute doesn't begin with COMPRESS_URL (which it is expecting to be the base url) it can't remove it and so raises an error.

How to solve this

It's difficult to know exactly what is going wrong without seeing the whole project, however the below might help:

  1. If you updated your AWS_S3_ENDPOINT_URL to match your AWS_S3_CUSTOM_DOMAIN. This would in turn update your STATIC_URL (which I suspect is affecting the filenames in your files), and will also update your COMPRESS_URL. I think this might fix it.

  2. Alternatively, just update COMPRESS_URL to use the correct url (although I think this needs to match STATIC_URL with your current setup).

  3. If any (link, rel or src) urls in files that are being compressed are hard-coded they need to be updated to match your COMPRESS_URL.

user_75499070

Django Compressor has to be configured for remote storage. Section 2.5.1 of the Django Compressor Documentation covers S3 and explains that you need to set

AWS_S3_CUSTOM_DOMAIN = '%s.s3.amazonaws.com' % AWS_STORAGE_BUCKET_NAME
COMPRESS_URL = f'https://{AWS_S3_CUSTOM_DOMAIN}/'

and

COMPRESS_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

in settings.py. Before I could get my app working, I still had to debug further such that I had to run collectstatic locally so that my_project/static matched my S3 storage. Then, I had to set

STATICFILES_LOCATION = ''

The latter two bugs may be unrelated issues, but I thought I should mention them.

user_126712800
Related