Prelude
None of the above worked for me.
Turns out I should have checked my notes from before, as this has been a recurring issue here with a project that uses Xunit:
Make sure all installed Xunit.xxxxxxxxx packages have the EXACT SAME version.
The problem occurs while/after having updated my NuGet packages automatically, which will, at the time of writing, install Xunit 2.4.1 (and several other xunit packages at version 2.4.1) plus xunit.runner.visualstudio at version 2.4.3 (!)
While nothing untoward is reported during this NuGet update, the result is a permanently failing build, where Fact and Assert are suddenly unknown, e.g.
Error CS0246: The type or namespace name 'Fact' could not be found (are you missing a using directive or an assembly reference?) Imazen.Test.Webp
The fix
What did work out for me was to go and revert that xunit.runner.visualstudio update, re-installing 2.4.1.
BTW, in Visual Studio, this would then look something like this (after the revert/re-install):

Note that the package manager there is hinting that an update is available and as soon as you apply that 2.4.3 update again, in any way, you're back to square one: a curiously failing build.
The key to the solution is to have all installed xunit packages with the same version. -- if only a few have updates available on NuGet, wait until all xunit packages are available for that same version.
Postscript
Don't know why this is so finicky, as I've only observed this brittle behaviour with xunit.*, but this is what has worked earlier this year (I had forgotten) and now had happen to me again, with the same outcome: the mandatory revert of a NuGet xunit package update.