Expire a view-cache in Django?

Viewed 24072

The @cache_page decorator is awesome. But for my blog I would like to keep a page in cache until someone comments on a post. This sounds like a great idea as people rarely comment so keeping the pages in memcached while nobody comments would be great. I'm thinking that someone must have had this problem before? And this is different than caching per url.

So a solution I'm thinking of is:

@cache_page( 60 * 15, "blog" );
def blog( request ) ...

And then I'd keep a list of all cache keys used for the blog view and then have way of expire the "blog" cache space. But I'm not super experienced with Django so I'm wondering if someone knows a better way of doing this?

15 Answers

This solution works for django versions before 1.7

Here's a solution I wrote to do just what you're talking about on some of my own projects:

def expire_view_cache(view_name, args=[], namespace=None, key_prefix=None):
    """
    This function allows you to invalidate any view-level cache. 
        view_name: view function you wish to invalidate or it's named url pattern
        args: any arguments passed to the view function
        namepace: optioal, if an application namespace is needed
        key prefix: for the @cache_page decorator for the function (if any)
    """
    from django.core.urlresolvers import reverse
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key
    from django.core.cache import cache
    # create a fake request object
    request = HttpRequest()
    # Loookup the request path:
    if namespace:
        view_name = namespace + ":" + view_name
    request.path = reverse(view_name, args=args)
    # get cache key, expire if the cached item exists:
    key = get_cache_key(request, key_prefix=key_prefix)
    if key:
        if cache.get(key):
            # Delete the cache entry.  
            #
            # Note that there is a possible race condition here, as another 
            # process / thread may have refreshed the cache between
            # the call to cache.get() above, and the cache.set(key, None) 
            # below.  This may lead to unexpected performance problems under 
            # severe load.
            cache.set(key, None, 0)
        return True
    return False

Django keys these caches of the view request, so what this does is creates a fake request object for the cached view, uses that to fetch the cache key, then expires it.

To use it in the way you're talking about, try something like:

from django.db.models.signals import post_save
from blog.models import Entry

def invalidate_blog_index(sender, **kwargs):
    expire_view_cache("blog")

post_save.connect(invalidate_portfolio_index, sender=Entry)

So basically, when ever a blog Entry object is saved, invalidate_blog_index is called and the cached view is expired. NB: haven't tested this extensively, but it's worked fine for me so far.

The cache_page decorator will use CacheMiddleware in the end which will generate a cache key based on the request (look at django.utils.cache.get_cache_key) and the key_prefix ("blog" in your case). Note that "blog" is only a prefix, not the whole cache key.

You can get notified via django's post_save signal when a comment is saved, then you can try to build the cache key for the appropriate page(s) and finally say cache.delete(key).

However this requires the cache_key, which is constructed with the request for the previously cached view. This request object is not available when a comment is saved. You could construct the cache key without the proper request object, but this construction happens in a function marked as private (_generate_cache_header_key), so you are not supposed to use this function directly. However, you could build an object that has a path attribute that is the same as for the original cached view and Django wouldn't notice, but I don't recommend that.

The cache_page decorator abstracts caching quite a bit for you and makes it hard to delete a certain cache object directly. You could make up your own keys and handle them in the same way, but this requires some more programming and is not as abstract as the cache_page decorator.

You will also have to delete multiple cache objects when your comments are displayed in multiple views (i.e. index page with comment counts and individual blog entry pages).

To sum up: Django does time based expiration of cache keys for you, but custom deletion of cache keys at the right time is more tricky.

Instead of using the cache page decorator, you could manually cache the blog post object (or similar) if there are no comments, and then when there's a first comment, re-cache the blog post object so that it's up to date (assuming the object has attributes that reference any comments), but then just let that cached data for the commented blog post expire and then no bother re-cacheing...

Most of the solutions above didn't work in our case because we use https. The source code for get_cache_key reveals that it uses request.get_absolute_uri() to generate the cache key.

The default HttpRequest class sets the scheme as http. Thus we need to override it to use https for our dummy request object.

This is the code that works fine for us :)

from django.core.cache import cache
from django.http import HttpRequest
from django.utils.cache import get_cache_key


class HttpsRequest(HttpRequest):
    @property
    def scheme(self):
        return "https"


def invalidate_cache_page(
    path,
    query_params=None,
    method="GET",
):
    request = HttpsRequest()

    # meta information can be checked from error logs
    request.META = {
        "SERVER_NAME": "www.yourwebsite.com",
        "SERVER_PORT": "443",
        "QUERY_STRING": query_params,
    }

    request.path = path
    key = get_cache_key(request, method=method)
    if cache.has_key(key):
        cache.delete(key)

Now I can use this utility function to invalidate the cache from any of our views:

page = reverse('url_name', kwargs={'id': obj.id})
invalidate_cache_page(path)

I struggled with a similar situation and here is the solution I came up with, I started it on an earlier version of Django but it is currently in use on version 2.0.3.

First issue: when you set things to be cached in Django, it sets headers so that downstream caches -- including the browser cache -- cache your page.

To override that, you need to set middleware. I cribbed this from elsewhere on StackOverflow, but can't find it at the moment. In appname/middleware.py:

from django.utils.cache import add_never_cache_headers


class Disable(object):

    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        response = self.get_response(request)
        add_never_cache_headers(response)
        return response

Then in settings.py, to MIDDLEWARE, add:

'appname.middleware.downstream_caching.Disable',

Keep in mind that this approach completely disables downstream caching, which may not be what you want.

Finally, I added to my views.py:

def expire_page(request, path=None, query_string=None, method='GET'):
    """
    :param request: "real" request, or at least one providing the same scheme, host, and port as what you want to expire
    :param path: The path you want to expire, if not the path on the request
    :param query_string: The query string you want to expire, as opposed to the path on the request
    :param method: the HTTP method for the page, if not GET
    :return: None
    """
    if query_string is not None:
        request.META['QUERY_STRING'] = query_string
    if path is not None:
        request.path = path
    request.method = method

    # get_raw_uri and method show, as of this writing, everything used in the cache key
    # print('req uri: {} method: {}'.format(request.get_raw_uri(), request.method))
    key = get_cache_key(request)
    if key in cache:
        cache.delete(key)

I didn't like having to pass in a request object, but as of this writing, it provides the scheme/protocol, host, and port for the request, pretty much any request object for your site/app will do, as long as you pass in the path and query string.

One more updated version of Duncan's answer: had to figure out correct meta fields: (tested on Django 1.9.8)

def invalidate_cache(path=''):
    import socket
    from django.core.cache import cache
    from django.http import HttpRequest
    from django.utils.cache import get_cache_key

    request = HttpRequest()
    domain = 'www.yourdomain.com'
    request.META = {'SERVER_NAME': socket.gethostname(), 'SERVER_PORT':8000, "HTTP_HOST": domain, 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, br'}
    request.LANGUAGE_CODE = 'en-us'
    request.path = path

    try:
        cache_key = get_cache_key(request)
        if cache_key :
            if cache.has_key(cache_key):
                cache.delete(cache_key)
                return (True, 'successfully invalidated')
            else:
                return (False, 'cache_key does not exist in cache')
        else:
            raise ValueError('failed to create cache_key')
    except (ValueError, Exception) as e:            
        return (False, e)
Related