Connect to TLS-enabled Elasticache Redis from Airflow Celery Executor

user_1872639 0
Viewed 1412

I am currently using Celery Executor on Airflow 1.10. My broker is AWS Elasticache Redis (v. 5.0.6). How can I enable Encryption in-transit?

According to Airflow source code, ssl_keyfile, ssl_certfile and ssl_ca_certs are required.

elif 'redis://' in broker_url:
    broker_use_ssl = {'ssl_keyfile': conf.get('celery', 'SSL_KEY'),
                      'ssl_certfile': conf.get('celery', 'SSL_CERT'),
                      'ssl_ca_certs': conf.get('celery', 'SSL_CACERT'),
                      'ssl_cert_reqs': ssl.CERT_REQUIRED}

https://github.com/apache/airflow/blob/1.10.10/airflow/config_templates/default_celery.py#L68-L72

But Elasticache Redis does not provide those TLS certificate. The official document only explain a solution for redis-cli, which uses TLS tunneling by stunnel.

https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html#connect-tls

Does Airflow Celery Executor support Encryption in-transit on Elasticache Redis? If so, how can we implement it?

amazon-elasticache airflow redis amazon-web-services
1 Answers

The following airflow config works with elasticache and redis ssl note the extra s on the rediss

broker_url = rediss://:password@host:port/db?ssl_cert_reqs=required
broker_use_ssl = { "ssl_cert_reqs": ssl.CERT_REQUIRED }

ssl_active = False
ssl_key =
ssl_cert =
ssl_cacert =
user_2371290
Related