How are people doing metadata management in large Azure data architectures? I'm particularly interested in how people are managing and versioning schemas for schema-on-read data architectures. Azure Data Catalog does not provide schema management. ADF does not provide a metadata store for use outside of ADF, and there does not seem to be a solution for hosting an external metadata store for use within ADF (e.g., reading a list of datasets from a Hive metastore). I understand you can use external metastores from HDInsight, but this seems to miss most of the ADF capabilities. The AWS Glue data catalog seems to be the closest thing to what I'm envisioning (and how you can use it across multiple AWS data services). I would expect Microsoft is working on something similar.
The options seem to be:
- Develop a custom metadata management solution and push to ADF and Hive metastores as needed
- Master the metadata in a Hive metastore and extract to ADF datasets
- Master the metadata as ADF datasets and push to Hive metastores
Question: How are people managing schemas in large Azure data architectures (e.g., multiple ADFs, data lakes, Spark clusters)?