Connect Databricks cluster with local machine (AWS)

user_10924136 0
Viewed 97

I want to connect to Databricks cluster (AWS) from my local machine but I want to execute the entire code in the cluster. With Databricks Connect only the spark code is executed in the cluster. I'm looking for alternative solution. SSH interpreter or something similar to that. I work with PyCharm (IDE).

pyspark databricks amazon-web-services
1 Answers

I would go with such a approach (but you need to write small script for your IDE):

  • you commit to some brunch in git (like staging)
  • your IDE executes databricks cli command "databricks repos update" which will perform pull
  • your IDE executes databricks cli job command to run notebook from repo

Databricks cli can be executed as a Rest Api, bash/cmd or can be imported as sdk to programming language

user_109729590
Related