In Celery I can use groups to keep track of a large number of tasks, but I have to have all the tasks up front to create the group:
job = group(my_function.s(link) for link in links)
result = job.apply_async()
I have a potentially large list of items I retrieve from a paginated and rate-limited API and I want to start processing those items as soon as I get the first page, without having to wait to get to the last page. Is this possible with groups or with other mechanisms provided by Celery?
Why I'd like to use groups is that they provide a way to see progress (how many of the tasks have finished or have failed).