How to avoid jar conflicts in a databricks workspace with multiple clusters and developers working in parallel?

We are working in an environment where multiple developers upload jars to a Databricks cluster with the following configuration:

DBR: 7.3 LTS
Operating System: Ubuntu 18.04.5 LTS
Java: Zulu 8.48.0.53-CA-linux64 (build 1.8.0_265-b11)
Scala: 2.12.10
Python: 3.7.5
R: R version 3.6.3 (2020-02-29)
Delta Lake: 0.7.0
Build tool: Maven

Below is our typical workflow:

STEP 0:

Build version 1 of the jar (DemoSparkProject-1.0-SNAPSHOT.jar) with the following object:

object EntryObjectOne {

  def main(args:Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("BatchApp")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    println("EntryObjectOne: This object is from 1.0 SNAPSHOT JAR")

    val df: DataFrame = Seq(
      (1,"A","2021-01-01"),
      (2,"B","2021-02-01"),
      (3,"C","2021-02-01")
    ).toDF("id","value", "date")

    df.show(false)
  }
}

STEP 1:

Uninstall the old jar(s) from the cluster, and keep pushing new changes in subsequent versions with small changes to the logic. Hence, we push jars with versions 2.0-SNAPSHOT, 3.0-SNAPSHOT etc.

At a point in time, when we push the same object with the following code in the jar say (DemoSparkProject-4.0-SNAPSHOT.jar):

object EntryObjectOne {

  def main(args:Array[String]): Unit = {
    val spark = SparkSession.builder()
      .appName("BatchApp")
      .master("local[*]")
      .getOrCreate()

    import spark.implicits._

    println("EntryObjectOne: This object is from 4.0 SNAPSHOT JAR")

    val df: DataFrame = Seq(
      (1,"A","2021-01-01"),
      (2,"B","2021-02-01"),
      (3,"C","2021-02-01")
    ).toDF("id","value", "date")

    df.show(false)
  }
}

When we import this object in the notebook and run the main function we still get the old snapshot version jar println statement (EntryObjectOne: This object is from 1.0 SNAPSHOT JAR). This forces us from running a delete on the dbfs:/FileStore/jars/* and restarting the cluster and pushing the latest snapshot again to make it work.

In essence when I run sc.listJars() the active jar in the driver is the latest 4.0-SNAPSHOT jar. Yet, I still see the logic from old snapshot jars even though they are not installed on the cluster at runtime.

Resolutions we tried/implemented:

• We tried using the maven shade plugin, but unfortunately, Scala does not support it. (details here).
• We delete the old jars from dbfs:/FileStore/jars/* and restart the cluster and install the new jars regularly. This works, but a better approach can definitely help. (details here).
• Changing the classpath manually and building the jar with different groupId using maven also helps. But with lots of objects and developers working in parallel, it is difficult to keep track of these changes.

Is this the right way of working with multiple jar versions in DataBricks? If there is a better way to handle this version conflict issue in DataBricks it will help us a lot.