When connecting to Databricks cluster from local IDE, I take that only spark-related commands are executed in remote mode (on cluster). How about single-node operations such as scikit-learn, to_pandas. If these functions only use local machine, the resource pool would be tiny. How to also utilize remote driver for execution of single-node operations ?
Use remote driver with Databricks Connect
Viewed 82
1 Answers
It's not possible by design of the Databricks Connect - in it, the local machine is always a Spark driver, and worker nodes of Databricks cluster are used as Spark executors. So all local operations like .collect will bring data to your machine & will run locally.
You may need to look onto dbx tool from Databricks Labs - it recently got dbx sync command that allows to automatically sync code changes to the Databricks repo, so you can write code in the IDE, and run it in the Databricks notebook, so in this case it will use driver from the Databricks cluster. (It won't allow to interactively debug code, but at least you can get code executed in the cloud, not on your machine)
Related