Databricks - an automated way to run cells several times by getting each time another input

tigrul 0
Viewed 24

everyone!

I need to run several cells in Databricks for each file I have. The input is an imagepath and for that imagepath there are like 40 cells that take 10 mins to run all of them. I have to run the code for 50 imagepaths. How can I automate this in Databricks in python? Is this even possible?

It is like this: get file and then run everything from below.

automation databricks python
1 Answers

You can use dbutils notebook API that allows you to run a notebook and pass widget parameters to it. It can be run in a loop like this:

imagepaths = [...]
for ip in imagepaths:
   dbutils.notebook.run("notebook", 0, {"imagepath": ip })

Note that it will run them sequentially. It would be nice to run the computation in parallel, but I'm not aware of a way to orchestrate it as Databricks job. You probably would need to do it externally with something like ADF or Airflow.

Kombajn zbożowy0
Related