You have a few options depending on what you want to do in the validator function and where you want to put the code that manages the dialog stack.

Option 1: return false

Your first opportunity to pop dialogs off the stack will be in the validator function itself, like I mentioned in the comments.

if (promptContext.AttemptCount > 3)
{
    var dc = await BotUtil.Dialogs.CreateContextAsync(promptContext.Context, cancellationToken);
    await dc.CancelAllDialogsAsync(cancellationToken);
    return false;
}

You were right to be apprehensive about this, because this actually can cause problems if you don't do it correctly. The SDK does not expect you to manipulate the dialog stack within a validator function, and so you need to be aware of what happens when the validator function returns and act accordingly.

Option 1.1: send an activity

You can see in the source code that a prompt will try to reprompt without checking to see if the prompt is still on the dialog stack:

if (!dc.Context.Responded)
{
    await OnPromptAsync(dc.Context, state, options, true, cancellationToken).ConfigureAwait(false);
}

This means that even if you clear the dialog stack inside your validator function, the prompt will still try to reprompt after that when you return false. We don't want that to happen because the dialog has already been cancelled, and if the bot asks a question that it won't be accepting answers to then that will look bad and confuse the user. However, this source code does provide a hint about how to avoid reprompting. It will only reprompt if TurnContext.Responded is false. You can set it to true by sending an activity.

Option 1.1.1: send a message activity

It makes sense to let the user know that they've used up all their attempts, and if you send the user such a message in your validator function then you won't have to worry about any unwanted automatic reprompts:

await promptContext.Context.SendActivityAsync("Cancelling all dialogs...");

Option 1.1.2: send an event activity

If you don't want to display an actual message to the user, you can send an invisible event activity that won't get rendered in the conversation. This will still set TurnContext.Responded to true:

await promptContext.Context.SendActivityAsync(new Activity(ActivityTypes.Event));

Option 1.2: nullify the prompt

We may not need to avoid having the prompt call its OnPromptAsync if the specific prompt type allows a way to avoid reprompting inside OnPromptAsync. Again having a look at the source code but this time in TextPrompt.cs, we can see where OnPromptAsync does its reprompting:

if (isRetry && options.RetryPrompt != null)
{
    await turnContext.SendActivityAsync(options.RetryPrompt, cancellationToken).ConfigureAwait(false);
}
else if (options.Prompt != null)
{
    await turnContext.SendActivityAsync(options.Prompt, cancellationToken).ConfigureAwait(false);
}

So if we don't want to send any activities to the user (visible or otherwise), we can stop a text prompt from reprompting simply by setting both its Prompt and RetryPrompt properties to null:

promptContext.Options.Prompt = null;
promptContext.Options.RetryPrompt = null;

Option 2: return true

The second opportunity to cancel dialogs as we move up the call stack from the validator function is in the next waterfall step, like you mentioned in your question. This may be your best option because it's the least hacky: it doesn't depend on any special understanding of the internal SDK code that could be subject to change. In this case your whole validator function could be as simple as this:

private Task<bool> ValidateAsync(PromptValidatorContext<string> promptContext, CancellationToken cancellationToken)
{
    if (promptContext.AttemptCount > 3 || IsCorrectPassword(promptContext.Context.Activity.Text))
    {
        // valid user input
        // or continue to next step anyway because of too many attempts
        return Task.FromResult(true);
    }

    // invalid user input
    // when there haven't been too many attempts
    return Task.FromResult(false);
}

Note that we're using a method called IsCorrectPassword to determine if the password is correct. This is important because this option depends on reusing that functionality in the next waterfall step. You had mentioned needing to save information in TurnState but this is unnecessary since everything we need to know is already in the turn context. The validation is based on the activity's text, so we can just validate that same text again in the next step.

Option 2.1: use WaterfallStepContext.Context.Activity.Text

The text that the user entered will still be available to you in WaterfallStepContext.Context.Activity.Text so your next waterfall step could look like this:

async (stepContext, cancellationToken) =>
{
    if (IsCorrectPassword(stepContext.Context.Activity.Text))
    {
        return await stepContext.NextAsync(null, cancellationToken);
    }
    else
    {
        await stepContext.Context.SendActivityAsync("Cancelling all dialogs...");
        return await stepContext.CancelAllDialogsAsync(cancellationToken);
    }
},

Option 2.2: use WaterfallStepContext.Result

Waterfall step contexts have a builtin Result property that refers to the result of the previous step. In the case of a text prompt, it will be the string returned by that prompt. You can use it like this:

if (IsCorrectPassword((string)stepContext.Result))

Option 3: throw an exception

Going further up the call stack, you can handle things in the message handler that originally called DialogContext.ContinueDialogAsync by throwing an exception in your validator function, like CameronL mentioned in the deleted portion of their answer. While it's generally considered bad practice to use exceptions to trigger intentional code paths, this does closely resemble how retry limits worked in Bot Builder v3, which you mentioned wanting to replicate.

Option 3.1: use the base Exception type

You can throw just an ordinary exception. To make it easier to tell this exception apart from other exceptions when you catch it, you can optionally include some metadata in the exception's Source property:

if (promptContext.AttemptCount > 3)
{
    throw new Exception(BotUtil.TooManyAttemptsMessage);
}

Then you can catch it like this:

try
{
    await dc.ContinueDialogAsync(cancellationToken);
}
catch (Exception ex)
{
    if (ex.Message == BotUtil.TooManyAttemptsMessage)
    {
        await turnContext.SendActivityAsync("Cancelling all dialogs...");
        await dc.CancelAllDialogsAsync(cancellationToken);
    }
    else
    {
        throw ex;
    }
}

Option 3.2: use a derived exception type

If you define your own exception type, you can use that to only catch this specific exception.

public class TooManyAttemptsException : Exception

You can throw it like this:

throw new TooManyAttemptsException();

Then you can catch it like this:

try
{
    await dc.ContinueDialogAsync(cancellationToken);
}
catch (TooManyAttemptsException)
{
    await turnContext.SendActivityAsync("Cancelling all dialogs...");
    await dc.CancelAllDialogsAsync(cancellationToken);
}

Declare a flag variable in user state class and update the flag inside the if block:

if (promptContext.AttemptCount > 3)
{
   \\fetch user state object
    \\update flag here
    return true;
}

After returning true you will be taken to the next dialog in waterfall step, where you can check the flag value, display an appropriate message and terminate dialog flow. You can refer to microsoft docs to know how to use the User state data

The prompt validator context object is a more specific object only concerned with passing or failing the validator.

** removed incorrect answer **

You can create a class with a WaterfallStep and a PromptValidator. That class would (i) handle the logic to exit the PromptValidator and (ii) handle the logic to cancel/end/proceed the dialog after that. This solution is category of Kyle Delaney answer which returns true in the PromptValidator.

I called that class WaterfallStepValidation:

private readonly Func<string, Task<bool>> _validator;
private readonly int _retryCount;
private bool _isInputValid = false;

public WaterfallStepValidation(Func<string, Task<bool>> validator, int retryCount)
{
    _validator = validator;
    _retryCount = retryCount;
}

public async Task<DialogTurnResult> CheckValidInputStepAsync(WaterfallStepContext stepContext, CancellationToken cancellationToken)
{
    if (!_isInputValid)
    {
        await stepContext.Context.SendActivityAsync("Could not proceed...");
        // Here you could also end all dialogs or just proceed to the next step
        return await stepContext.EndDialogAsync(false);
    }
    return await stepContext.NextAsync(stepContext.Result, cancellationToken);
}

public async Task<bool> PromptValidatorAsync(PromptValidatorContext<string> promptContext, CancellationToken cancellationToken)
{
    _isInputValid = await _validator(promptContext.Recognized.Value);

    if (!_isInputValid && promptContext.AttemptCount >= _retryCount)
    {
        _isInputValid = false;
        return true;
    }

    return _isInputValid;
}

And then you call it like this:

var ageStepValidation = new WaterfallStepValidation(AgeValidator, retryCount: 3);
AddDialog(new TextPrompt("AgeTextPromptId", ageStepValidation.PromptValidatorAsync));

var waterfallSteps = new List<WaterfallStep>()
{
    PromptNameStepAsync,
    PromptAgeStepAsync,
    ageStepValidation.CheckValidInputStepAsync,
    PromptChoicesStepAsync
};
AddDialog(new WaterfallDialog(nameof(WaterfallDialog), waterfallSteps));

I think this is an elegant workaround for this problem. The weak points of this aproch are:

• you have to place CheckValidInputStepAsync right after the step with the validation
• is still a kind of hack, since the PromptValidator returns true if fails the validation

However, the strong points are:

• it doesn't make big hacks, just that return true
• most of the logic is encapsulated in the WaterfallStepValidation class