When I scan Azure databricks with Azure Purview with the specification of any of my clusters only a subset of databases are returned together with only a few of the tables! The scan ends successfully but it is incomplete. Out of more than 20 databases only 3 databases are detected!
Setup:
I have an Azure databricks workspace. I am trying to scan the databricks by Azure purview. Following this link I have registered a hive database and used the databricks URL to finish the registration. It is shown below:
As to scan databricks, purview required to have a Self Hosted Integration Runtime (SHIR) by following this link I have prepared a Windows (Windows Server 2022 Datacenter Azure Edition) VM to scan. All the drivers and JDK 11 are installed and the service has the right access. Here you can see the SHIR is up and running and detected by Azure purview:
In my Azure databricks, I have 7 clusters and more than 20 databases. But when I scan it by Azure Purview, it only detects 14 objects which include only 3 databases, and only a few tables of those databases. Some of these databases have more than 30 tables, but as shown below even detected databases are not scanned fully.
In data acatalog, in browse assets, databricks is detected after successful scan:
Here you can see only 3 databases are detected!
While there are more than 20 databases:
Please note that these databases are returned by using the same cluster that I used in purview! So here in Azure databricks that cluster has access to everything, but when I use its properties to define the new scan in Azure purview only 3 databases are returned and even they are incomplete and not containing most of their tables!
I have 7 clusters, and I tried each of them to scan the databricks, but still each scan return again the same 14 objects. Here you see my clusters:
One other relevant point is how I make the scan: I have read the hive-site.xml by launching the web terminal in Azure databricks, and put the relevant values in the new scan. This is the mapping of the hive-site.xml and the required fields of the scan:
- Metastore JDBC Driver Location: location of drivers on SHIR. In my case a folder on desktop of the VM.
- Metastore JDBC Driver Class: Value of javax.jdo.option.ConnectionDriverName. In my case: org.mariadb.jdbc.Driver
- Metastore JDBC URL: Value of javax.jdo.option.ConnectionURL
- Metastore database name: Extracting this value from javax.jdo.option.ConnectionURL. This contains <database_name>:<port_number> like jdbc:mariadb://<db_name>.mysql.database.azure.com:3306
And here is the scan connection which was as expected successful, otherwise it wouldn't have shown the scan result as shown previously above.
Reading the hive-site.xml, I see that all my clusters have mariadb as javax.jdo.option.ConnectionDriverName. And the scan was full.
I would like to know why the scan doesn't detect all objects and how I can resolve the issue.