Is it possible to loop over a prefect.Parameter?

user_1803908 0
Viewed 1359

I have the following flow:

with Flow("My flow") as flow:
    urls = Parameter("urls", default=['url1', 'url2'])
    for url in urls:
        result = get_url(urls)

When I run this by flow.run(), I get the following error:

TypeError: 'Parameter' object is not iterable

Is it possible to iterate over it and if yes, how?

prefect etl python-3.x python
1 Answers

This is a very common stumbling block for people new to deferred computing paradigms.

Prefect Parameters (and more generally, any Prefect task) represent units of work that will run in the future. When you build a flow, you are specifying dependencies between units of work that will run later. This is a powerful abstraction because it lets you reason about certain properties of your workflow before you ever run it.

The caveat is that you need to distinguish between what is available at build time (prior to ever running the Flow) and what is available at run time (during the context of a flow run). In your case, you are using a piece of knowledge that Prefect has no way of knowing about which is that your Parameter is going to return something iterable. Moreover, even if Prefect could infer that the parameter output will be iterable, it has no way of knowing how many elements will be returned (which could be different with every run of the flow!).

Given all of that, Prefect handles these "deferred iterations" with a concept called "mapping": https://docs.prefect.io/core/concepts/mapping.html

In your case, you could refactor your code to:

with Flow("My flow") as flow:
    urls = Parameter("urls", default=['url1', 'url2'])
    result = get_url.map(urls)

which will dynamically spawn copies of get_url for each value in the parameter's return value.

user_16178870
Related