Redis connection pool with Lambda

Question

I am trying to use Redis with lambda functions. How can I use the connection pooling in the lamdba if it does not maintain state. Is it ok to always have the lambda make new connections to redis? Or can I use the connection pool example I pasted below? Where in my Handle func I will do conn := pool.Get(). Not sure what approach I should take. Any help.

func Handle(ctx context.context, req events.APIGatewayWebsocketProxyRequest)(interface{},error){

  //make new redis connection


}

or

func newPool(addr string) *redis.Pool {
  return &redis.Pool{
    MaxIdle: 3,
    IdleTimeout: 240 * time.Second,
    // Dial or DialContext must be set. When both are set, DialContext takes precedence over Dial.
    Dial: func () (redis.Conn, error) { return redis.Dial("tcp", addr) },
  }
}

var (
  pool *redis.Pool
  redisServer = flag.String("redisServer", ":6379", "")
)

func main() {
  flag.Parse()
  pool = newPool(*redisServer)
  ...
}

user_1564064 · Answer

Although not the perfect solution (like e.g. AWS RDS Proxy), creating the connection pool outside the lambda's handler seems to be your best bet. The actual technique depends on your language/environment as you can also lazily initialize your connection pool and save it to a class variable, singleton, etc. In your example, use a global pool variable from anywhere in the lambda handler function.

var (
  pool *redis.Pool
  redisServer = flag.String("redisServer", ":6379", "")
)

func newPool(addr string) *redis.Pool {
  ...
}

func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
  conn := pool.Get()
  defer conn.Close()
  err = conn.Send("ZINCRBY", ...)
  if err != nil {
    return err
  }
  ...
}

func main() {
  flag.Parse()
  pool = newPool(*redisServer)
  lambda.Start(HandleRequest)
}

Check this Redislabs article for some more info.

AWS lambda's processing model is a rather simple one. As long as the requests are coming (lambda is being called from API gateway, Kinesis stream, etc.), lambda's lifecycle management code will keep the lambda alive as it synchronously loops over the batches of requests. When this single-threaded solution can't keep up with the incoming processing requests (by constantly checking on some latency metrics), lambda's lifecycle manager will create more lambdas. So, in a sense, when there's a lot to process, you will have your lambda(s) up for some time, and that's when you want to be conservative with your server's free connections anyhow.

Redis connection pool with Lambda

1 Answers