Does it Make Sense to Dynamically Add EC2 Instance Private IPs to Django's CACHE LOCACTION Setting in an Elastic Beanstalk Environment with NGINX?

Viewed 29

I'm using Elastic Beanstalk with load balancing and I've set a minimum number of instances to 2. So I have two instances with two different private IPs.

What I'm trying to achieve is to add Memcached caching locations dynamically to Django's CACHES setting based on the EC2 instances that exist in an environment.

My Primary Question

Since Elastic Beanstalk uses NGINX (in my case), do I need to dynamically add the instance IPs to Django's CACHES location? Or would setting 'LOCATION': '127.0.0.1:11211', be sufficient for all running EC2 instances?

Secondary Question Does NGINX centralize the cache for all running instances?

My NGINX primary config:

(staging) [ec2-user@ip-172-31-9-242 conf.d]$ cat elasticbeanstalk/00_application.conf 
location / {
    proxy_pass          http://127.0.0.1:8000;
    proxy_http_version  1.1;

    proxy_set_header    Connection          $connection_upgrade;
    proxy_set_header    Upgrade             $http_upgrade;
    proxy_set_header    Host                $host;
    proxy_set_header    X-Real-IP           $remote_addr;
    proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
}

The Django documentation says the same cache can be shared across multiple servers - Memcached caching:

One excellent feature of Memcached is its ability to share a cache over multiple servers. This means you can run Memcached daemons on multiple machines, and the program will treat the group of machines as a single cache, without the need to duplicate cache values on each machine. To take advantage of this feature, include all server addresses in LOCATION, either as a semicolon or comma delimited string, or as a list.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
        'LOCATION': [
            '172.19.26.240:11211',
            '172.19.26.242:11211',
        ]
    }
}

I made a class using boto3 to get a list of EC2 instances:

class MemcachedLocations(object):

    MEMCACHED_PORT = "11211"
    ACCESS_KEY = os.environ.get('AWS_S3_ACCESS_KEY_ID')
    SECRET_KEY = os.environ.get('AWS_S3_SECRET_KEY_ID')
    REGION = 'us-west-2'

    @cached_property
    def get(self):

        client = boto3.client('ec2', region_name=self.REGION, aws_access_key_id=self.ACCESS_KEY,
                              aws_secret_access_key=self.SECRET_KEY)

        response = client.describe_instances()

        private_ips_list = []
        for reservation in response['Reservations']:
            for instance in reservation['Instances']:
                private_ip = instance['PrivateIpAddress']
                private_ip = f"{private_ip}:{self.MEMCACHED_PORT}"
                private_ips_list.append(private_ip)

        return private_ips_list

It works as expected when deployed:

>>> from django.conf import settings
>>> settings.CACHES
{'default': 
    {'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
     'LOCATION': ['172.##.##.###:11211', '172.##.##.###:11211']
    }
}

but when I test it in SSH:

from django.core.cache import cache
cache.set('ping', 'pong', 600)
cache.get('ping')
>>> Returns None but should return 'pong' if working properly

Sidenote: I already know about AWS Elasticache. I already know how to implement it successfully. I'd like to not use it, if possible.

0 Answers
Related