MVC Authorize Attribute Work without Loging off

I have a controller class that only a specific Active Directory Group should be able to access.

[Authorize(Roles = @"Domain\GroupName")]
public class AdminToolsController : Controller
{
    ...
}

Now as I am testing.. I am currently out of the group.. but if I add myself.. and I try to access anything in this controller I still get asked to login and my credentials do not work. However.. if I add myself.. then logoff.. then log back on.. then try to access anything in this controller it recognizes me and allows me access.

Is there anyway to do this instantaneously? Meaning, can I add myself to the group and successfully access any of the methods inside the controller without having to logoff and log back on?

UPDATE

I have edited Camilo Terevinto answer below. Their answer for some reason.. whenever I added or removed my self from the specific group.. that group would not be a part of the variable groups.

Here is my update:

public class AuthorizeByActiveDirectoryGroups : AuthorizeAttribute
{
    protected override bool AuthorizeCore(HttpContextBase httpContext)
    {
        var roles = Roles.Split(',');
        using (var domainContext = new PrincipalContext(ContextType.Domain, "domainname"))
        {
            var user = httpContext.User.Identity.Name;
            using (var domainUser = UserPrincipal.FindByIdentity(domainContext, httpContext.User.Identity.Name))
            {
                var adgroup = GroupPrincipal.FindByIdentity(domainContext, "Domain\\GroupName");
                bool member = domainUser.IsMemberOf(adgroup);
                var groups = domainUser.GetAuthorizationGroups();
                return member;
            }
        }
    }
}