I have multiple SparkSubmitOperators launched in Cluster mode. Currently, I have to use "yarn logs" command to fetch the logs manually.
Ideally, I would like to have the logs fetched in the Airflow task logs directly. I could not find a satisfying solution... any help appreciated