Adding IDPs after startup

I am currently running multiple instances of IdentityServer4 on dotnetcore 2.1 (migrating to 3.0 soon) and have a list of SAML IDPs stored in the database. I am able to initialize them at startup using the following code as an extension on the authenticationBuilder.

        //add IDPs at startup - saml providers comes from DB
        foreach (var samlProvider in samlProviders)
        {               
            authenticationBuilder.AddSaml2(samlProvider.Scheme, samlProvider.Name, options =>
            {                   
                var entityId = new EntityId(my.EntityId);
                options.SignInScheme = IdentityServerConstants.ExternalCookieAuthenticationScheme;
                options.SPOptions.EntityId = entityId;
                options.SPOptions.ModulePath = samlProvider.ModulePath;

                if (samlProvider.MinimumSigningAlgorithm != null)
                    options.SPOptions.MinIncomingSigningAlgorithm = samlProvider.MinimumSigningAlgorithm;

                var idp = new IdentityProvider(entityId, options.SPOptions)
                {
                    Binding = Saml2BindingType.HttpRedirect,
                    LoadMetadata = true
                };

                if (samlProvider.MetaDataLocation != null)
                    idp.MetadataLocation = samlProvider.MetaDataLocation;

                options.IdentityProviders.Add(idp);
            });
        }

I would like to be able to add or remove IDPs after startup based on the contents of the aforementioned SamlProviders table. I have created a service that does this and is called by a background worker since we are running multiple servers and need to guarantee each server updates.

        // get list of new and removed providers
        foreach (var provider in removedProviders)
        {
            _schemeProvider.RemoveScheme(provider.Name);
            // Do I need to remove previous options from the SAML cache?
        }
        foreach (var provider in providerList)
        {

            Saml2Options newOptions = BuildSaml2Options(provider.EntityId, provider.ModulePath, provider.MinimumSigningAlgorithm, provider.MetaDataLocation);

            if (await _schemeProvider.GetSchemeAsync(provider.Scheme) == null)
                _schemeProvider.AddScheme(new AuthenticationScheme(provider.Scheme, provider.Name, typeof(Saml2Handler)));

            //How can I add saml options for the new authentication scheme here?
        }

Is it possible to add new Identity providers dynamically like this? If so is it possible to back into where the settings are stored and update them for each provider or add/remove them for provider changes without recycling the apps to get configure services running?