I have to use circleCI to deploy jobs in Databricks. So far I have scoured the internet trying to find a piece of information about it, Couldn’t find any. Can anyone help me with this please!?
1 Answers
It's really a very broad question - you need to clarify what pieces need to be deployed, etc.
Databricks has a number of integrations described in the documentation & also in a number of blog posts like this - you just need to adapt the described code to CircleCI-specific syntax - the overall structure is the same.
Usually, all integration is done either Databricks REST API or using databricks-cli that allows to upload notebooks, create/update or trigger jobs, etc.
Specifically for CI/CD purposes you can look onto dbx project, especially to dbx init command that generates a skeleton of a project pre-configured for different CI/CD systems - it doesn't have CircleCI in the list, but it could be easily adapted.
Related