How do I read task parameters in a Databricks Job?

user_1350796 0
Viewed 168

The job is running a Python file, not a notebook. All the examples I can find online use the notebook utility which Databricks says is unreliable inside of executors.

databricks
1 Answers

Figured it out - when calling from the Jobs API, you have to include

"spark_python_task":{"python_file":"dbfs:/FileStore/my_script.py", "parameters": ["param1", "param2"]}

and when calling from the UI (run with new paramaters) you need to include

{"spark_python_task": ["param1","param2"]}

Each time, for reading you read with sys.argv. (sys.argv[1] and sys.argv[2] in this example).

user_13507960
Related