Defer block in Task causes "Main actor-isolated property 'someProperty' can not be mutated from the main actor"

I'm writing a cancelable async function which does a lot of work with many early returns interspersed if the task has been canceled. I need to perform some post processing tear-down work when the task successfully runs to completion or is canceled. To accomplish that, I tried to put this logic in a defer block but this reveals an odd error message:

Main actor-isolated property 'isPerformingAsyncWork' can not be mutated from the main actor

Is this not a direct contradiction, saying this property can only be modified on the main actor and you cannot mutate it from the main actor? How do you resolve this or otherwise execute some code when the task finishes/gets canceled?

Here's some sample code to demonstrate the problem:

class ViewController: UIViewController {
    
    var isPerformingAsyncWork = false

    override func viewDidAppear(_ animated: Bool) {
        super.viewDidAppear(animated)
        
        performAsyncWork()
    }
    
    func performAsyncWork() {
        guard !isPerformingAsyncWork else { return }
        
        let alert = UIAlertController(title: "Working on it…", message: nil, preferredStyle: .alert)
        
        let task = Task {
            defer {
                //called upon task completion or cancelation
                
                //FIXME: Main actor-isolated property 'isPerformingAsyncWork' can not be mutated from the main actor
                isPerformingAsyncWork = false
            }
            
            isPerformingAsyncWork = true
            
            let url = URL(string: "https://hws.dev/user-favorites.json")!
            let (data, _) = try await URLSession.shared.data(from: url)
            
            guard !Task.isCancelled else { return }
            
            let values = try JSONDecoder().decode([Int].self, from: data)
            
            guard !Task.isCancelled else { return }
            
            //more async work here, more isCancelled checks, etc...

            alert.presentingViewController?.dismiss(animated: true)
        }
        
        alert.addAction(UIAlertAction(title: "Cancel", style: .cancel) { action in
            task.cancel()
        })
        present(alert, animated: true)
    }

}