Sensibly set MUI error status with react-hook-form

The accepted way (based on looking at lots of answers here) to set error state for MUI components with RHF seems to be something like

<Controller
  name='username'
  control={control}
  defaultValue=''
  render={({
    field: { onChange, value, name, ref },
    fieldState: { error }
  }) => (
    <TextField
      name={name}
      value={value}
      onChange={onChange}
      InputRef={ref}
      error={!!error}
      helperText={error?.message}
    />
  )}
/>

The problem with this, as is, is that if (for example) the field is required, then it starts in an initial state of being error highlighted and displaying the helper text. Things can be improved by making sure that the user has at least entered some data before we set the highlighting with isDirty:

    fieldState: { isDirty, error }
    ...
      error={isDirty && !!error}
      helperText={isDirty && error?.message}

That gets us most of the way there, but if someone tries to submit the form, but hasn't touched one of the required fields, it doesn't highlight that field.

I played around with various combinations of onBlur, formState.isSubmitting, and formState.isSubmitted, but couldn't get it to behave such that

• Initial state shows no errors
• User input which does not validate shows an error after the user leaves the field
• All non-validating entries are highlighted when a user clicks on submit (but before some data-heavy / api-based submit action actually takes place)

How does one achieve this? (Or, alternatively, if my three-bullet rubric is for some sensible UI reason not the right rubric, what should I care about and how do I achieve that?