setJobDescription(description) allows to put a name to a job (the action) we are doing.
Often, for a Spark session, a single call is enough for me : a collect(), a show(), and I'm done.
But for stages, it's different. A the time Spark plan to do a transformation somewhat "moving" it creates one. filter(...), map(...), groupBy(...)...
I would like to put also stages names with setCallSite(description).
But there's a trap. Here's my job :
session.setJobDescription("accounting balance")
// Many transformations that are creating stages
// ---------------------------------------------
session.sparkContext().setCallSite("Reading account data for cities");
Dataset<Row> comptes = load(/* ...stage : load from store... */)
session.sparkContext().setCallSite("Checking validity of records");
comptes = comptes.filter((FilterFunction<Row>)compte -> {
/* stage : checking validity of records */
});
session.sparkContext().setCallSite("Adding account descriptions from nomenclatures");
comptes = comptes.join( /*... stage : adding account descriptions from accounting plan */)
session.sparkContext().setCallSite("Confrontation with city code"));
comptes = comptes.join(/*....stage joining geographical code ...*/)
comptes = comptes.filter(/*....stage : removing unknown cities ...*/)
session.sparkContext().setCallSite("Cumulation of root accounts");
comptes = /* many stages behind : it's a method */
// Job done.
comptes.show(); // the action
Here's what I see on Spark UI, all the time :
Accounting balance
Cumulation of root accounts
It's normal : Spark generated its plan, its DAG, by going over my code a first time, during the analysis phase.
And it executed all the session.sparkContext().setCallSite(...) it contains, until the last one.
This, before having started any stage.
If I was really willing the stage name to be displayed at due time, theoretically, I should have to write something like that, instead :
comptes = comptes.filter((FilterFunction<Row>)compte -> {
session.sparkContext().setCallSite("Checking validity of records");
/* stage : checking validity of records */
});
But it's not possible, and more : it would be very dirty. Setting the same description at each iteration of the filter loop !!!
What's the way to change the call site description at the very time a stage is starting, and not during the analysis phase ?