Owin Self-Host WebApi Windows Authentication and Anonymous

I have a self-hosted Owin WebAPI. I want to protect a few routes with authentication. The majority of the routes should be accessible anonymously. I have succesfully implemented Windows-Auth, but now I get 401 - Unauthorized when trying to access the routes marked with [AllowAnonymous] when accessing them anonymously. If I call the method with valid credentials all works fine.

The perfect solution would be to allow anonymous by default and only require credentials when the action has the [Authorize] attribute.

Owin config

public void Configuration(IAppBuilder appBuilder)
{
    // Enable Windows Authentification
    HttpListener listener = (HttpListener)appBuilder.Properties["System.Net.HttpListener"];
    listener.AuthenticationSchemes = AuthenticationSchemes.IntegratedWindowsAuthentication;

    HttpConfiguration config = new HttpConfiguration();
    config.MapHttpAttributeRoutes();

    appBuilder.Use(typeof(WinAuthMiddleware));
    appBuilder.UseWebApi(config);
}

WinAuth OwinMiddleware

public class WinAuthMiddleware : OwinMiddleware
{
    public WinAuthMiddleware(OwinMiddleware next) : base(next) {}
    public async override Task Invoke(IOwinContext context)
    {
        WindowsPrincipal user = context.Request.User as WindowsPrincipal;
        //..
    }
}

An example Action

public class ValuesController : ApiController
{      
    [AllowAnonymous] // attribute gets ignored
    [Route("Demo")]
    [HttpGet]
    public string Get()
    {
        //..
    }
}