Execution_date in DockerOperator airflow

user_9530178 0
Viewed 169

I have a docker container that parse audio files in text files via docker container that is to be mounted to the folder where these audiofiles are. I want to create an Airflow dag, that will to this task automatically. There is my plan

  1. I download audiofiles to the folder,that is names as execution_date,i.e. /path/{{ds}}
  2. I run docker container through docker operator with volumes = ['/path/{{ds}}'] The problem is that docker operator tries to find folder /path/{{ds}} without transforming {{ds}} into date and taking it as a string. How can i fix it? Thatks for helping
dockeroperator airflow docker
1 Answers

volumes isn't a templated parameter.

You can overcome this by creating MyDockerOperator as:

from airflow.providers.docker.operators.docker import DockerOperator

class MyDockerOperator(DockerOperator):
     template_fields = ('volumes') + DockerOperator.template_fields

By doing so MyDockerOperator has all the functionality of DockerOperator with the additional support of volumes as templated parameter.

user_146244090
Related