Getting adal4j/AuthenticationException while connecting to Azure SQL from Google Dataproc Cluster

Please hear me out.

I am able to successfully write to Azure SQL server table using Apache Spark connector in Dataproc cluster. But then while testing I found sometime that the SQL server database in not available and because of this the write operation fails.

So, I decided to add an exception logic to check if the database is available or not using the following code before writing to Azure DB,

    var connection: Connection = null
    val url = "jdbc:sqlserver://***.database.windows.net:1433;databaseName=some_db;encrypt=true;trustServerCertificate=true;authentication=ActiveDirectoryPassword"
    val user = "xxxx"
    val password = ******
    try
          {
            Class.forName("com.microsoft.sqlserver.jdbc.SQLServerDriver")
            connection = DriverManager.getConnection(url, user, password)
            log.info(s"Successfully established connection on attempt: "+attempts)
            return
          }

This code works fine in my local machine. But while running the same in Dataproc I am getting the below error,

Exception in thread "main" java.lang.NoClassDefFoundError: com/microsoft/aad/adal4j/AuthenticationException

I already went though this similar issue discussed in SO. But it's bit different for me as the Apache Spark connector to write to SQL server is working fine. I just want to have the code DriverManager.getConnection(url, user, password) work to test the connection before my write.