Underline text font in UWP's TextBox?

You can try adding the border to the default style, but I think that's gonna be a tough one, you can use RichEditBox for now while working on overriding the default style through creating a new style resource, I understand that RichEditBox is not designed for accepting inputs from the user but I think it wouldn't hurt that much with the performance https://docs.microsoft.com/en-us/windows/uwp/design/controls-and-patterns/text-controls

Am I missing something basic? Or should I forget TextBox and use RichEditBox (which would be a massive overkill for my purposes)?

You could use the core text APIs in the Windows.UI.Text.Core namespace to make your own custom input control.

You just need to use some general UWP control(e.g, textblock, stackpanel, border etc) to make a input control (you could call it your own custom textbox) which has underline when you input text in it.

Here's a UWP official Custom Edit Control sample. I just specified the TextDecorations property value for the TextBlock in 'CustomEditControl.xaml', then, I achieved your target.

<StackPanel x:Name="BorderPanel" BorderThickness="4" Background="White">
        <StackPanel x:Name="ContentPanel" Orientation="Horizontal" HorizontalAlignment="Left">
            <TextBlock x:Name="BeforeSelectionText" Foreground="Black" TextDecorations="Underline"/>
            <TextBlock x:Name="CaretText" Text="&#xe12b;" Foreground="Blue" FontFamily="Segoe UI Symbol"/>
            <Border Background="Blue">
                <TextBlock x:Name="SelectionText" Foreground="White"/>
            </Border>
            <TextBlock x:Name="AfterSelectionText" Foreground="Black"/>
        </StackPanel>
</StackPanel>