Run same Databricks notebook for different arguments concurrently?

user_1212412 0
Viewed 1304

The following code (not mine) is able to run NotebookA and NotebookB concurrently. I need some help to figure out how to pass multiple arguments to the same notebooks.

I want to pass this list of arguments to each notebook:

args = {}
args["arg1"] = "some value"
args["arg2"] = "another value"

If I wanted to pass the arguments above to each of the running notebooks, what will I need to amend in the code below?

Here is the working code:

from multiprocessing.pool import ThreadPool
pool = ThreadPool(10)
inputs = [("NotebookA", "NotebookB") ]

run_in_parallel = lambda x: dbutils.notebook.run(x, 1800) 

from concurrent.futures import ThreadPoolExecutor, wait 
pool = ThreadPoolExecutor(3) 
results = [] 
with ThreadPoolExecutor(3) as pool:
  for x in inputs: 
    results.extend(pool.map(run_in_parallel, list(x)))
pyspark databricks
1 Answers

The dbutils.notebook.run accepts the 3rd argument as well, this is a map of parameters (see documentation for more details). So in your case, you'll need to change definition of the run_in_parallel to something like this:

run_in_parallel = lambda x: dbutils.notebook.run(x, 1800, args)

and the rest of the code should be the same.

If you'll want to pass different arguments to different notebooks, then you'll need to have a list of tuples, and pass this list to a map, like this:

data = [('notebook 1', {'arg1':'abc'}), ('notebook2', {'arg1': 'def', 'arg2': 'jkl'})]
...
run_in_parallel = lambda x: dbutils.notebook.run(x[0], 1800, x[1]) 
with ThreadPoolExecutor(3) as pool:
  results.extend(pool.map(run_in_parallel, data))
Alex_Ott_186270
Related