Why do my assemblies need to be loaded in a specific order?

I am writing a simple plugin and stumbled upon contractType.IsAssignableFrom(pluginType) returning different results depending on the order of loading.

Calling IsAssignableFromon the Plugin returns True as expected.
But if I load the Contract assembly before loading the Plugin, IsAssignableFrom on the Plugin returns False.

I am running Win10 and dotnet4.7 but I doubt that has any relevancy.

Code

[TestMethod]
public void SimplyLoadingPlugin_Succeeds()
{
    var plugin = Assembly.LoadFrom(PluginPathFilename);
    var res = typeof(Contract).IsAssignableFrom(plugin.GetExportedTypes().Single());

    Assert.IsTrue(res); // Succeeds.
}

[TestMethod]
public void LoadingContractAndThenPlugin_Fails()
{
    var contract = Assembly.LoadFrom(ContractPathFilename);
    var plugin = Assembly.LoadFrom(PluginPathFilename);
    var res = typeof(Contract).IsAssignableFrom(plugin.GetExportedTypes().Single());

    Assert.IsTrue(res); // Fails.
}

To make it harder to test:
If i run the LoadingContractAndThenPlugin_Fails test by itself is fails. But If i run the tests together it is dependent on order. Running SimplyLoadingPlugin_Succeeds first and LoadingContractAndThenPlugin_Fails last, makes both tests green but running them in the reverse order makes both fail.
So somehow the very loading of Contract before Plugin messes up something for me.
I can se nothing related in the GAC.

Below are all files needed. The paths in the probably have to be updated.
4 project with one file in each. 1 solution.

Contract.cs (a library project)

public abstract class Contract
{
    public abstract int Version { get; set; }
}

Plugin.cs (a library project)

public class Plugin : Contract
{
    public override int Version { get; set; }
}

Tests.cs (a test project)

[TestClass]
public class Tests
{
    private const string PluginPath = @"C:\DATA\Projekt\LoadFromOrder\Plugin\bin\Debug";
    private string PluginPathFilename = Path.Combine(PluginPath, "Plugin.dll");
    private string ContractPathFilename = Path.Combine(PluginPath, "Contract.dll");

    [TestMethod]
    public void SimplyLoadingPlugin_Succeeds()
    {
        var plugin = Assembly.LoadFrom(PluginPathFilename);
        var res = typeof(Contract).IsAssignableFrom(plugin.GetExportedTypes().Single());

        Assert.IsTrue(res); // Succeeds.
    }

    [TestMethod]
    public void LoadingContractAndThenPlugin_Fails()
    {
        var contract = Assembly.LoadFrom(ContractPathFilename);
        var plugin = Assembly.LoadFrom(PluginPathFilename);
        var res = typeof(Contract).IsAssignableFrom(plugin.GetExportedTypes().Single());

        Assert.IsTrue(res); // Fails.
    }

    // BEGIN ---- Update. ----
    [TestMethod]
    public void LoadingPluginFromTestProject_Succeeds()
    {
        var contract = Assembly.LoadFrom(
            @"C:\DATA\Projekt\LoadFromOrder\TestProject\bin\Debug\Contract.dll");
        var plugin = Assembly.LoadFrom(PluginPathFilename);
        var res = typeof(Contract.Contract).IsAssignableFrom(plugin.GetExportedTypes().Single());

        Assert.IsTrue(res); // Succeeds.
    }
    // END ---- Update. ----

}

Program.cs (a console project)

class Program
{
    static void Main(string[] args)
    {
        var tests = new Tests();
        try
        {
            System.Console.WriteLine("Press A for Success and B for Fail.");
            switch (System.Console.ReadKey(true).Key)
            {
                case ConsoleKey.A:
                    tests.SimplyLoadingPlugin_Succeeds();
                    break;
                case ConsoleKey.B:
                    tests.LoadingContractAndThenPlugin_Fails();
                    break;
            }
            System.Console.WriteLine("SUCCESS");
        }
        catch (Exception exc)
        {
            System.Console.WriteLine($"FAIL: {exc.Message}");
        }
    }
}