I frequently do analysis of the DAG of my spark job while it is running. But, it is annoying to have to sit and watch the application while it is running in order to see the DAG.
So, I tried to view the DAg using this thing called the spark history-server, which I know should help me see past jobs. I'm easily able to access port 18080, and I can see the history server UI.
But, it doesn't show me any information related to the spark program's execution. I know I have the history server running, because when I do sudo service --status-all I see
spark history-server is running [ OK ]
So I already tried what this question suggested: here.
I think this is because I'm running spark on YARN, and it can only use one resource manager at a time? maybe?
So, how do I see the spark execution DAG, *after* a job has finished? and more specifically, when running YARN as my resource manager?