We are running WSO2 APIM 2.6.0 (yes, I know, we will upgrade soon :)) as an API Manager/Gateway. We have added a JWT Token Generator (extending JWTGenerator). The idea is that when the request comes in the gateway adds a JWT containing info about the user and environment and then our API layer behind that reads that and uses that info.
The Problem
The JWT token is supposed to be cached. This is working in some environments, but in others it's like the cache isn't cleared: the same JWT is used for the whole lifetime of the bearer token! This means that after a while it's invalid and the API layer is effectively down until the bearer token expires.
How does the caching work? Is there some process that cleans it, that I can look at? What could be causing this?
Details
My understanding is that the gateway can cache the tokens, to avoid regenerating them. We have therefore defined the expiry time of the JWT to be twice the cache time. The cache time is at default 15 mins, so I think this means the token has between 15 and 30 mins remaining once it reaches the API layer. My understanding is this (example times to illustrate):
- 1000 Bearer token token issued
- 1000 First request with bearer token
- First request so cache empty - generate JWT with expiry of 1030 and cache it until 1015
- API layer gets token with expiry 1030 (now + 30 mins)
- 1010 Another request with bearer token
- Token still in cache so used
- API Layer gets token with expiry 1030 (now + 20 mins)
- 1014.59
- Token still in cache so used
- API Layer gets token with expiry 1030 (now + 15 mins)
- 1015
- Cache has expired so new token generated with expiry of 1045 and cached until 1030
- API layer gets token with expiry 1045 (now + 30 mins)
- 1100 Bearer token expires and new one request
- 1100 First request with new bearer token - new bearer token so cache empty and starts from the top
This is what I experience in most environments. (Our first version had an issue as we had used the cache time instead of twice it, so the JWT at 1014.59 only had one second left. We fixed by doubling, tested and it worked.)
It still works in most environments, but in some environments it is using the original JWT for the entire lifetime of the bearer token! This means the API is unavailable for 30 minutes out of every hour! (If you are using client_credentials you cannot force-invalidate the bearer token)
I'm pretty sure that the config is the same between the environments that work and don't - the whole thing is managed by Ansible and both are recent installations. The api-manager.xml fields are identical except for some URLs that are environment specific
Relevant parts of the api-manager.xml are:
<JWTConfiguration>
<!-- Enable/Disable JWT generation. Default is false. -->
<!-- EnableJWTGeneration>false</EnableJWTGeneration-->
<!-- Name of the security context header to be added to the validated requests. -->
<JWTHeader>X-JWT-Assertion</JWTHeader>
<!-- Fully qualified name of the class that will retrieve additional user claims
to be appended to the JWT. If not specified no claims will be appended.If user wants to add all user claims in the
jwt token, he needs to enable this parameter.
The DefaultClaimsRetriever class adds user claims from the default carbon user store. -->
<!--ClaimsRetrieverImplClass>org.wso2.carbon.apimgt.impl.token.DefaultClaimsRetriever</ClaimsRetrieverImplClass-->
<!-- The dialectURI under which the claimURIs that need to be appended to the
JWT are defined. Not used with custom ClaimsRetriever implementations. The
same value is used in the keys for appending the default properties to the
JWT. -->
<!--ConsumerDialectURI>http://wso2.org/claims</ConsumerDialectURI-->
<!-- Signature algorithm. Accepts "SHA256withRSA" or "NONE". To disable signing explicitly specify "NONE". -->
<!--SignatureAlgorithm>SHA256withRSA</SignatureAlgorithm-->
<!-- This parameter specifies which implementation should be used for generating the Token. JWTGenerator is the
default implementation provided. -->
<JWTGeneratorImpl>com.carus.api.gateway.JwtTokenGenerator</JWTGeneratorImpl>
<!-- This parameter specifies which implementation should be used for generating the Token. For URL safe JWT
Token generation the implementation is provided in URLSafeJWTGenerator -->
<!--<JWTGeneratorImpl>org.wso2.carbon.apimgt.keymgt.token.URLSafeJWTGenerator</JWTGeneratorImpl>-->
<EnableJWTGeneration>true</EnableJWTGeneration>
<ClaimsRetrieverImplClass>org.wso2.carbon.apimgt.impl.token.DefaultClaimsRetriever</ClaimsRetrieverImplClass>
<ConsumerDialectURI>https://carus.com/claims</ConsumerDialectURI>
<SignatureAlgorithm>NONE</SignatureAlgorithm>
<RemoveUserNameFromJWTForApplicationToken>false</RemoveUserNameFromJWTForApplicationToken>
</JWTConfiguration>
and
<CacheConfigurations>
<!-- Enable/Disable token caching at the Gateway-->
<EnableGatewayTokenCache>true</EnableGatewayTokenCache>
<!-- Enable/Disable API resource caching at the Gateway-->
<EnableGatewayResourceCache>true</EnableGatewayResourceCache>
<!-- Enable/Disable API key validation information caching at key-management server -->
<EnableKeyManagerTokenCache>false</EnableKeyManagerTokenCache>
<!-- This parameter specifies whether Recently Added APIs will be loaded from the cache or not.
If there are multiple API modification during a short time period, better to disable cache. -->
<EnableRecentlyAddedAPICache>false</EnableRecentlyAddedAPICache>
<!-- This parameter specifies whether scopes are taken from cache or not. If you are modifying application
subscriptions frequently, modifying the user roles frequently or updating the subscribed APIs frequently, it
is better to turn-off this cache-->
<EnableScopeCache>true</EnableScopeCache>
<!-- This indicates whether the role cache need to enabled in the publisher. If this is disabled, there will
be a call to key manager to all the calls to API publisher APIs. It is highly recommended to enable this
cache. However, if the system is in a state, where the role addition and deletion happens seamlessly, the
cache will be in in-valid state.-->
<EnablePublisherRoleCache>true</EnablePublisherRoleCache>
<!-- JWT claims Cache expiry in seconds -->
<!--JWTClaimCacheExpiry>900</JWTClaimCacheExpiry-->
<!-- Expiry time for the apim key mgt validation info cache -->
<!--TokenCacheExpiry>900</TokenCacheExpiry-->
<!-- This parameter specifies the expiration time of the TagCache. TagCache will
only be created when this element is uncommented. When the specified
time duration gets elapsed ,tag cache will get re-generated. -->
<!--TagCacheDuration>120000</TagCacheDuration-->
<!-- JWT Claim cache can be disabled only if below config <EnableJWTClaimCache> is set as 'false'. The default
value is 'true'.
Other than this config, also <EnableGatewayTokenCache> and <EnableKeyManagerTokenCache> should have been set
as 'false' to completely disable JWTClaimCache.
-->
<EnableJWTClaimCache>true</EnableJWTClaimCache>
</CacheConfigurations>
The initial part of our JWT token generator (including calculating the expiry) is
@Override
public Map<String, String> populateStandardClaims(final TokenValidationContext validationContext)
throws APIManagementException {
APIKeyValidationInfoDTO context = validationContext.getValidationInfoDTO();
if (LOGGER.isDebugEnabled()) {
LOGGER.debug("Validation context: <" + context + ">.");
}
String dialect = getDialectURI();
// Matches what the parent class uses, but doubled
// This is because getTTL() defaults to cache timeout, when caching active,
// so when that nears the token has almost expired
long expireIn = System.currentTimeMillis() + getTTL() * 2000L;
I have confirmed (by logging the token itself out) that the token is refreshed after 15 mins in one environment and not in the other.
I have also confirmed all the machines have the correct time and time zone, both the OS and also in MariaDB
Any ideas? Where to look next? I can't create a proper MCVE as I can't repeat it at will. Well, I can in some environments, but can't see what makes them different from others where it doesn't happen, so I need help finding this!
I can work-around by ensuring the JWT-lifetime is more than the Bearer token lifetime, but I count that as a work-around, not a solution.