why did i get redis command timeout when using brpop comand whose timeout option value(50 seconds)is less than redis command timeout(200s) setting?

Viewed 355

I'm using redis list as a distributed blocking queue. On the client side, I use the following code:

public String tryAquire(String appName, long timeout, TimeUnit timeUnit){
  return String.valueOf(redisTemplate.opsForList.rightPop(getKey(appName), timeout, timeUnint)); 
}

It uses brpop command internal and the timeout value is set less than 50 seconds. This service works fine for about two weeks, until recent last 2 days I got a few exceptions of this:

org.springframework.QueryTimeoutException:Redis command time out;
nest exception is io.lettuce.core.RedisCommandException:
Command time out after 200 seconds(s)

This exception appears 1 or 2 times a day for about 2000 requests per day and after this exception the server still works fine and subsequent requests' cost time become normal, but the request which throws this exception would cost more than 200 seconds and that is a very bad case.

This time out value (200 seconds) fits my lettuce client side command timeout setting.

lettuce client setting

However, for the tryAquire(appName, timeout, timeUnit) method, the maximum blocking time is set less than 50 seconds. Thus this command should not cost time which is so much longer than 50s, because after 50 seconds if there is no elements in redis list it should just return null rather than keep waiting. It seems like there is no network issue for there is no socket related exception in log and after the redis command timeout exception the subsequent request is executed successfully.

1 Answers

Just in case someone encounteres the same problem. Long story short:The NAT mapping expires and the broken network link is awared of by the application after max tcp retries has been reached.

This service is deployed at a cloud environment which uses a customed SDN. The NAT mapping expires if there is no activity on a tcp connection for some time. However both tcp client and server can't be aware of that and the client would keep trying sending data even if the NAT mapping has expired, until some tcp retries max limit configuraion is reached.

A simple solution: Just set the redis server "tcp-keepalive" configuration value smaller than the NAT mapping expire time or use any other heartbeat mechanism if you don't want to change the redis server setting.

Related