Prefect 2 Dask: submit Not Consuming Resources

Zap 0
Viewed 44

Goal: I am trying to Get tasks created by Prefect 2 for Dask to consume the GPU resource to prevent over working.

Checked: I setup my dask-workers with a resource of GPU=1 each. I have checked the dashboard and confirmed they all have GPU as 1.

Problem: When running my task via Prefect 2 I don’t see the resource being marked as consumed and then eventually each worker gets over worked and crashes.

This is what I am doing in prefect from my understand of the documentation in https://prefecthq.github.io/prefect-dask/:

import requests
from prefect import flow, task, get_run_logger
from prefect_dask.task_runners import DaskTaskRunner
import dask

@task
def UpscaleFrames(FramesToUpscale):
    # Do some work with CUDA
    return

@flow(task_runner=DaskTaskRunner(address="tcp://tower:8786"))
def Upscale():
    for file in GetVideoFiles("/videos"):
        while (frames_found):
            FramesToUpscale = GetFramesToUpscale()
            with dask.annotate(resources={'GPU': 1}):
                    UpscaleFrames.submit(FramesToUpscale)

Version Info:

Version:             2.3.2
API version:         0.8.0
Python version:      3.10.6
Git commit:          6e931ee9
Built:               Tue, Sep 6, 2022 12:36 PM
OS/Arch:             linux/x86_64
Profile:             default
Server type:         hosted
distributed-computing dask distributed prefect python
1 Answers

Could you share a full example of your flow to see how you apply that? Also, what is your Prefect version? Printing the output of prefect version would be helpful.

If you run on self-hosted Orion, the likely problem is networking -- this is something that would be easier to set up for a PoC with Cloud (there is a free tier so you can just sign up to troubleshoot: https://app.prefect.cloud/).

If nothing else works, it might help to create a GitHub issue on the Collection repo https://github.com/PrefectHQ/prefect-dask/issues describing all the steps you've taken so far and what doesn't work as expected

Anna Geller0
Related