Handling Task Exceptions - custom TaskScheduler

I am writing a custom FIFO-queued thread-limited task scheduler.

1. I would like to be able to guarantee that unhandled task exceptions bubble up or escalate immediately, and proceed to end the process.

2. I would also like to be able to Task.Wait() and handle an AggregateException that might pop out, without causing the process to end.

Can these two requirements be met simultaneously? Am I thinking about this the wrong way?

Further information

The crux of the issue is that I don't always want to Task.Wait() on a task to complete.

After executing the task on my custom task scheduler, I know that unhandled task exceptions will eventually escalate when the Task is cleaned up by the GC.

If you do not wait on a task that propagates an exception, or access its Exception property, the exception is escalated according to the .NET exception policy when the task is garbage-collected.

But who knows when that will happen in the environment in which the application executes - this can't be determined at design time.

To me, this means that unhandled exceptions in tasks can potentially go undetected - forever. That leaves a bad taste in my mouth.

If I want to ensure that unhandled task exceptions are escalated immediately, I can do the following in my task scheduler after the task is executed:

while ( taskOnQueue )
{
    /// dequeue a task

    TryExecuteTask(task);

    if (task.IsFaulted) 
    { 
        throw task.Exception.Flatten(); 
    }
}

But by doing this I basically guarantee that the exception will always terminate the process, regardless of how it may be handled by catching an AggregateException at Task.Wait() (or in a TaskScheduler.UnobservedException event handler).

Given these options - do I really have to choose one or the other?