Updating jar job on databricks

user_18238032 0
Viewed 283

I have a shared cluster which is used by more than several jobs on databricks. the update of the jar corresponding to the job is not used when I launch the execution of the job, on cluster, I see that it uses an old version of the jar.

to clarify, I publish the jar through API 2.0 in databricks.

my question why when i start the execution of my Job, the execution on the cluster always uses an old version. Thank you for you helping

databricks apache-spark
1 Answers

Old jar will be removed from the cluster only when it's terminated. If you have a shared cluster that never terminates, then it doesn't happen. This a limitation not of the Databricks but Java that can't unload classes that are already in use (or it's very hard to implement reliably).

For most of cases it's really not recommended to use shared cluster, for several reasons:

  • it costs significantly more (~4x)
  • tasks are affecting each other from performance point of view
  • there is a high probability of dependencies conflicts + inability of updating libraries without affecting other tasks
  • there is a kind of "garbage" collected on the driver nodes
  • ...

If you use shared cluster to get faster execution, I recommend to look onto Instance Pools, especially in combination of preloading of Databricks Runtime onto nodes in instance pool.

Alex_Ott_186270
Related