I am using Apache Airflow where in one of our DAG's task we are using Kubernetes Pod Operator. This is being done to execute one of our application process in a kubernetes pod. The Kubernetes pod operator works all good.Passing environmental variables via the pod operator works all good. However I am trying to pass an environment variable value from a Kubernetes ConfigMap it is not able to get the values from ConfigMap.
Code snippet is as below. In the code snippet please focus on the line 'SPARK_CONFIG': '{"valueFrom": {"configMapKeyRef": {"key": "endpoint","name": "spark-config"}}}'
Please find below the code snippet
pod_process_task = KubernetesPodOperator(
namespace=cons.K8_NAMESPACE,
image=cons.UNCOMPRESS_IMAGE_NAME,
config_file=cons.K8_CONFIG_FILE,
env_vars={
'FRT_ID': '{{ dag_run.conf["transaction_id"] }}',
'FILE_NAME': '{{ dag_run.conf["filename"]}}',
'FILE_PATH': '{{dag_run.conf["filepath"]}}' + "/" + '{{ dag_run.conf["filename"]}}',
'LOG_FILE': '{{ ti.xcom_pull(key="process_log_dict")["loglocation"] }}',
'SPARK_CONFIG': '{"valueFrom": {"configMapKeyRef": {"key": "endpoint","name": "spark-config"}}}'
},
name=create_pod_name(),
# name= 'integrator',
task_id="decrypt-951",
retries=3,
retry_delay=timedelta(seconds=60),
is_delete_operator_pod=True,
volumes=[volume-a,volume_for_configuration],
volume_mounts=[volume_mount_a,volume_mount_config],
resources=pod_resource_specification,
startup_timeout_seconds=cons.K8_POD_TIMEOUT,
get_logs=True,
on_failure_callback=log_failure_unzip_decrypt,
dag=dag
)
Then on trying to print the variable from the pod I am getting as below. Please note that the other ENV_VARIABLE values have been populated except for the one where I am trying to reference the configMap. Please find below the log output that we get to see while the K8-Pod is being spawned. [In the snippet below , please focus on the parameter 'name': 'SPARK_CONFIG' ]. The rest of the env_variables seems to have the value populated as what I have provided in the code snippet above with Jinga-templating.
'containers': [{'args': None,
'command': None,
'env': [{'name': 'FRT_ID',
'value': '20180902_01605',
'value_from': None},
{'name': 'FILE_NAME',
'value': 'transact_2018-09-02_0321_file_MAR.zip',
'value_from': None},
{'name': 'FILE_PATH',
'value': '/etc/data/app/trk0057.zip',
'value_from': None},
{'name': 'LOG_FILE',
'value': 'log-0057_2018-09.log',
'value_from': None},
{'name': 'SPARK_CONFIG',
'value': '{"valueFrom": {"configMapKeyRef": '
'{"key": "endpoint","name": '
'"spark-config"}}}',
'value_from': None}],
'env_from': None
...
...
...
...
The point is as to how do we pass the ConfigMap value as with the value_from while forming the environment variable in the Apache Airflow kubernetes pod operator