Remove a route with IOperationFilter in SwashBuckle

I am looking for a way to show/hide WebAPI routes in the Swagger documentation using SwashBuckle in a configurable way. Adding [ApiExplorerSettings(IgnoreApi = true)] will indeed hide the routes but I'd need to recompile every time I want that to change.

I have looked into creating an IOperationFilter to work with a custom Attribute that I defined. That way I can decorate the routes with a [SwaggerTag("MobileOnly")] and check the web.config or something to see if the route should be shown. The Attribute is defined as such:

public class SwaggerTagAttribute : Attribute
{
    public string[] Tags { get; private set; }

    public SwaggerTagAttribute(params string[] tags)
    {
        this.Tags = tags;
    }
}

The IOperationFilter that detects the attribute is defined and the IDocumentFilter that removes the path is defined here:

public class RemoveTaggedOperationsFilter : IOperationFilter, IDocumentFilter
{
    private List<string> TagsToHide;

    public RemoveTaggedOperationsFilter()
    {
        TagsToHide = ConfigurationManager.AppSettings["TagsToHide"].Split(',').ToList();
    }

    public void Apply(Operation operation, SchemaRegistry schemaRegistry, ApiDescription apiDescription)
    {
        var tags = apiDescription.ActionDescriptor
            .GetCustomAttributes<SwaggerTagAttribute>()
            .Select(t => t.Tags)
            .FirstOrDefault();

        if (tags != null && TagsToHide.Intersect(tags).Any())
        {
            operation.tags = new List<string> {"Remove Me "};
        }
    }

    public void Apply(SwaggerDocument swaggerDoc, SchemaRegistry schemaRegistry, IApiExplorer apiExplorer)
    {
        foreach (var value in swaggerDoc.paths.Values)
        {
            if (value.post != null && value.post.tags.Contains("Remove Me"))
                value.post = null;

            if (value.get != null && value.get.tags.Contains("Remove Me"))
                value.get = null;

            if (value.put != null && value.put.tags.Contains("Remove Me"))
                value.put = null;

            if (value.delete != null && value.delete.tags.Contains("Remove Me"))
                value.delete = null;
        }
    }
}

And registered as such:

 GlobalConfiguration.Configuration
            .EnableSwagger(c =>
                {
                    c.OperationFilter<RemoveTaggedOperationsFilter>();
                    c.DocumentFilter<RemoveTaggedOperationsFilter>();
                });

I feel that this is inefficient and hacky to tag something for removal later when I have access to it earlier. Is there any way that I just remove the route from the within IOperationFilter.Apply rather than wait for the IDocumentFilter and scan through it?