Prefect how to wait for external dependency

user_3186067 0
Viewed 1119

I have a prefect flow that I want to run if and when a specific file appears. With something like Luigi you would create an ExternalTask that outputs that file and then impose a dependence on it. What is the standard pattern for this in Prefect?

prefect etl pipeline python
1 Answers

There are a few options depending on your use case:

  • create a waiting task: in this case, you can write a root task for your flow that waits for the external dependency / condition to be met, and then returns. As long as the other tasks depend on this one, they won't run until this task completes.
  • use the GraphQL API: both Prefect Server and Cloud have a fully featured GraphQL API for performing many common actions with flows and runs. In this case, you can call create_flow_run whenever your external condition is met (possibly with Parameter values describing the condition) to create an ad-hoc run of your flow. For more discussion of this pattern, check out this stackoverflow question
user_16178870
Related