How to get Serilog to respect ILoggingBuilder SetMiniumLevel

I want to set the minimum log level programatically at runtime on a set of loggers. I am using Microsoft.Extensions.Logging and as well as Serilog (with Serilog.Extensions.Logging).

It seems that Serilog does not respect the log level set on the ILoggingBuilder:

var level = LogLevel.Debug;
Log.Logger = new LoggerConfiguration().WriteTo
    .Console(outputTemplate: "[Serilog {SourceContext} {Level:w4}] {Message:lj}{Exception}{NewLine}")
    .CreateLogger();

var loggerFactory = LoggerFactory.Create(builder =>
    builder.SetMinimumLevel(level)
        .AddSerilog()
        .AddSimpleConsole(options => { options.SingleLine = true;})
    );
var logger = loggerFactory.CreateLogger("LoggingTest");

logger.LogInformation("Info Message");
logger.LogDebug("Debug Message");

Produces:

[Serilog LoggingTest info] Info Message
info: LoggingTest[0] Info Message
dbug: LoggingTest[0] Debug Message

I could sync these by using .MinimumLevel.ControlledBy(levelSwitch) on Log.Logger a long with a mapping from Mircosoft to Serilog Log Levels, but I would then need to change both where ever one is set which isn't ideal.

Is there a way to get Serilog to respect the minimum log level set on the builder?