Define resource_defs in dagster job sensor

user_15581402 0
Viewed 300

I am trying to build sensor for the execution of pipeline/graph. The sensor would check on different intervals and executes the job containing different ops. Now the Job requires some resource_defs and config. In the offical documentation I don't see how I can define resource_defs for Job. A small hint would be great

Question : where or how do i define resource_defs in sensor ? Do I even have to define it ? its not mentioned in official documentation

https://docs.dagster.io/concepts/partitions-schedules-sensors/sensors

### defining Job 
@job(
resource_defs = {"some_API_Module": API_module , "db_Module" : db} ,
 config = {key : value } 
) 
def job_pipeline ():
    op_1 () ## API is used as required resource  
    op_2 () ##  db is used as required resource 

### defining sensor that triggers the Job
@sensor ( Job = job_pipeline) :
    ### some calculation 
    yield RunRequest(run_key = "" config = {key : value} )
dagster orchestration pipeline api python
1 Answers

It seems like you may be missing building your run config from within the sensor. The configuration you pass to a RunRequest should contain the resource configuration you want to run the job with, and will look exactly like the run configuration you'd configure from the launchpad. Something like:

### defining sensor that triggers the Job
@sensor ( Job = job_pipeline) :
    ### some calculation 
    run_config = {
            "ops": {
                "op1": {
                    "config": {
                        "key": "value"
                        },
                    }
                },
                "op2": {
                    "config": {
                        "key": "value",
                    }
                },
            },
            "resources": {
                "some_API_module": {
                    "config": {"key": "value"}
                  },
                "db": {
                    "config": {"key": "value"}
                 },
            },
        }
    yield RunRequest(run_key="<unique_value>", run_config=run_config)
user_62978000
Related