How to solve a Redis timeout on client side with StackExchange.Redis?

user_4200001 0
Viewed 23708

I'm using StackExchange.Redis client in a c# wcf application : I use only the synchronous command to get and set values. The problem is I have a timeout with this curious log :

Timeout performing EXISTS DataKey:50, 
inst: 1, queue: 1, qu: 0, qs: 1, qc: 0, wr: 0, wq: 0, in: 0, ar: 0, clientName: Machine1, 
serverEndpoint: redis-server:6381, keyHashSlot: 7984, 
IOCP: (Busy=1,Free=799,Min=8,Max=800), 
WORKER: (Busy=5,Free=795,Min=8,Max=800) 
(Please take a look at this article for some common client-side issues that can 
cause timeouts: http://stackexchange.github.io/StackExchange.Redis/Timeouts)

If I understand that correctly it means that my get value is queued because there is five worker threads ? Using netstat, I see that my application is opening two physical connections to the server . I have made sure to have enought threads available in my threadpool. In my connection settings I have a syncTimeout=3000... If I use the redis-cli, I could get the value of the key in 0.64 secondes.

Can anyone help please? What can I do? Does I have to use async in my code all the way or find another redis client lib ?

stackexchange.redis timeout c#
3 Answers

There are a few solutions. First of all they have a writeup on the github repo about the solutions they propose, found here.

What I've found though is just use the async functions as much as you can, it is only the synchronous functions that timeout. It seems to me that the timeouts are a design decision on their part based on the idea that they don't want to block your code if something goes wrong. I don't buy that, and so the work around I use is to use the async functions and then wait for the task. So instead of db.StringGet("thestring") I just do db.StringGetAsync("thestring").Result. That is, if I cannot use await for whatever reason.

You might want to look into using async/await as much as possible anyway. You might also want to use FireAndForget if you that is appropriate. You can also use ContinueWith. They talk about those solutions here.

user_14859950

From "WORKER: (Busy=5,Free=795,Min=8,Max=800)", I see that there are 5 Busy worker threads and a minimum of only 8 worker threads. I suspect that if you look at the whole set of errors that you received, you might find that Busy is greater than the Min count in some cases. This would indicate threadpool throttling. The link also provides some solutions on increasing the Min Thread counts for Worker and IOCP threads.

user_36839030

fixing RedisTimeout exception is always a challenge, there any many factors that cause timeout Exception,

in your log you can see for IOCP thread there are 1 busy threads and your system is configured to allow 8 minimum threads. so it's not the case because 1 is not > 8

once I had the same situation and I fixed timeout exceptions by increasing syncTimeout in the connection string, you could try to increase it

I also realized that number of open connections affect Redis performance, you can look at open connections by running this command 

info clients

I had around 90 open connections at the time that timeout exception was thrown, I restarted the server to reset open connection and timeout exceptions disappeared

user_45073940
Related