I'm using Elastic Beanstalk with load balancing and I've set a minimum number of instances to 2. So I have two instances with two different private IPs.
What I'm trying to achieve is to add Memcached caching locations dynamically to Django's CACHES setting based on the EC2 instances that exist in an environment.
My Primary Question
Since Elastic Beanstalk uses NGINX (in my case), do I need to dynamically add the instance IPs to Django's CACHES location? Or would setting 'LOCATION': '127.0.0.1:11211', be sufficient for all running EC2 instances?
Secondary Question Does NGINX centralize the cache for all running instances?
My NGINX primary config:
(staging) [ec2-user@ip-172-31-9-242 conf.d]$ cat elasticbeanstalk/00_application.conf
location / {
proxy_pass http://127.0.0.1:8000;
proxy_http_version 1.1;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
The Django documentation says the same cache can be shared across multiple servers - Memcached caching:
One excellent feature of Memcached is its ability to share a cache over multiple servers. This means you can run Memcached daemons on multiple machines, and the program will treat the group of machines as a single cache, without the need to duplicate cache values on each machine. To take advantage of this feature, include all server addresses in LOCATION, either as a semicolon or comma delimited string, or as a list.
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
'LOCATION': [
'172.19.26.240:11211',
'172.19.26.242:11211',
]
}
}
I made a class using boto3 to get a list of EC2 instances:
class MemcachedLocations(object):
MEMCACHED_PORT = "11211"
ACCESS_KEY = os.environ.get('AWS_S3_ACCESS_KEY_ID')
SECRET_KEY = os.environ.get('AWS_S3_SECRET_KEY_ID')
REGION = 'us-west-2'
@cached_property
def get(self):
client = boto3.client('ec2', region_name=self.REGION, aws_access_key_id=self.ACCESS_KEY,
aws_secret_access_key=self.SECRET_KEY)
response = client.describe_instances()
private_ips_list = []
for reservation in response['Reservations']:
for instance in reservation['Instances']:
private_ip = instance['PrivateIpAddress']
private_ip = f"{private_ip}:{self.MEMCACHED_PORT}"
private_ips_list.append(private_ip)
return private_ips_list
It works as expected when deployed:
>>> from django.conf import settings
>>> settings.CACHES
{'default':
{'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
'LOCATION': ['172.##.##.###:11211', '172.##.##.###:11211']
}
}
but when I test it in SSH:
from django.core.cache import cache
cache.set('ping', 'pong', 600)
cache.get('ping')
>>> Returns None but should return 'pong' if working properly
Sidenote: I already know about AWS Elasticache. I already know how to implement it successfully. I'd like to not use it, if possible.