I have a Spring boot app which talks to two different Redis clusters (on Amazon Elasticache). I'm using spring-data-redis 1.6.4. This is my code for the different Redis configurations:
@Configuration
public class RedisConfig {
@Bean
@Primary
public JedisConnectionFactory clusterAJedisConnectionFactory() {
JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
jedisConnectionFactory.setHostName(clusterAUrl);
jedisConnectionFactory.setPort(clusterAPort);
jedisConnectionFactory.setUsePool(true);
return jedisConnectionFactory;
}
@Bean
public JedisConnectionFactory clusterBJedisConnectionFactory() {
JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
jedisConnectionFactory.setHostName(clusterBUrl);
jedisConnectionFactory.setPort(clusterBPort);
jedisConnectionFactory.setUsePool(true);
return jedisConnectionFactory;
}
@Bean(name="clusterARedisTemplate")
public RedisTemplate<String, Object> clusterARedisTemplate() {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<String, Object>();
redisTemplate.setConnectionFactory(clusterAJedisConnectionFactory());
redisTemplate.setKeySerializer( new StringRedisSerializer() );
redisTemplate.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
redisTemplate.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
return redisTemplate;
}
@Bean(name="clusterBRedisTemplate")
public RedisTemplate<String, Object> clusterBRedisTemplate() {
RedisTemplate<String, Object> redisTemplate = new RedisTemplate<String, Object>();
redisTemplate.setConnectionFactory(clusterBJedisConnectionFactory());
redisTemplate.setKeySerializer( new StringRedisSerializer() );
redisTemplate.setHashValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
redisTemplate.setValueSerializer( new GenericToStringSerializer< Object >( Object.class ) );
return redisTemplate;
}
}
And then, in my code I have something like this for using it:
@Service
RedisService {
private final RedisConfig redisConfig;
private final ObjectMapper mapper;
@Autowired
public RedisCache(RedisConfig redisConfig, ObjectMapper mapper) {
this.redisConfig = redisConfig;
this.mapper = mapper;
}
@Async
public void saveValueInClusterA(String cacheKey, MyObject myObject) {
try {
String cacheValue = mapper.writeValueAsString(myObject);
redisConfig.clusterARedisTemplate().opsForValue().set(cacheKey, cacheValue, 1, TimeUnit.HOURS);
} catch ( Exception e ) {
LOGGER.error(...);
}
}
public MyObject getValueFromClusterA(String cacheKey) {
MyObject myObject = null;
try {
String cachedEntry = redisConfig.clusterARedisTemplate().opsForValue().get(cacheKey).toString();
myObject = mapper.readValue(cachedEntry, MyObject.class);
} catch ( Exception e ) {
LOGGER.error (...);
}
return myObject;
}
@Async
public void saveValueInClusterB(String cacheKey, MyObject myObject) {
try {
String cacheValue = mapper.writeValueAsString(myObject);
redisConfig.clusterBRedisTemplate().opsForValue().set(cacheKey, cacheValue, 1, TimeUnit.HOURS);
} catch ( Exception e ) {
LOGGER.error(...);
}
}
public MyObject getValueFromClusterB(String cacheKey) {
MyObject myObject = null;
try {
String cachedEntry = redisConfig.clusterBRedisTemplate().opsForValue().get(cacheKey).toString();
myObject = mapper.readValue(cachedEntry, MyObject.class);
} catch ( Exception e ) {
LOGGER.error (...);
}
return myObject;
}
}
This works fine under normal load. However, when I did a load test and took a thread dump, I saw that most of the threads are waiting on something like this:
"XNIO-2 task-973" #1547 prio=5 os_prio=0 tid=0x00007f472c41d800 nid=0x2d4e waiting on condition [0x00007f4680851000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000004ab53fb58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.commons.pool2.impl.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:583)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:442)
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
at redis.clients.util.Pool.getResource(Pool.java:48)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:99)
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:12)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:155)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:251)
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:58)
at org.springframework.data.redis.core.RedisConnectionUtils.doGetConnection(RedisConnectionUtils.java:128)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:91)
at org.springframework.data.redis.core.RedisConnectionUtils.getConnection(RedisConnectionUtils.java:78)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:178)
at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:153)
at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:86)
at org.springframework.data.redis.core.DefaultValueOperations.set(DefaultValueOperations.java:182)
at com.mypkg.services.RedisService.saveValueInClusterA(RedisService.java:97)
at com.mypkg.services.RedisService$$FastClassBySpringCGLIB$$aa4c9d31.invoke( )
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651)
at com.mypkg.services.RedisService$$EnhancerBySpringCGLIB$$a879b180.saveValueInClusterA( )
at com.mypkg.services.impl.MyImpl.method2(MyImpl.java:745)
at com.mypkg.services.impl.MyImpl.method1(MyImpl.java:419)
at sun.reflect.GeneratedMethodAccessor487.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:52)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:59)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
…..
…..
…..
…..
…..
…..
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:284)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:263)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:174)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:202)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:793)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Locked ownable synchronizers:
- <0x00000004b41253a0> (a java.util.concurrent.ThreadPoolExecutor$Worker)
I set the max pool size as 128, even though AWS console says there are only 35 connections max at any time to the redis cluster. What is going on here? Is my redis configuration wrong? Or do I need to release the connections after every use? I thought Redis template handles all those internally. Is the fact that I'm connecting to multiple redis clusters causing some issue?
Thanks.