We are using Rancher 2.2.13 and Kubernetes 1.13.12 in GKE. Our instance keeps flapping while being connected in Rancher. The agent logs show:
E0804 00:50:08.384154 6 request.go:853] Unexpected error when reading response body: context.deadlineExceededError{}
E0804 00:50:08.384223 6 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.Secret: Unexpected error context.deadlineExceededError{} when reading response body. Please retry.
E0804 00:50:08.385380 6 request.go:853] Unexpected error when reading response body: &http.httpError{err:"context deadline exceeded (Client.Timeout exceeded while reading body)", timeout:true}
E0804 00:50:08.385431 6 reflector.go:134] github.com/rancher/norman/controller/generic_controller.go:175: Failed to list *v1.ConfigMap: Unexpected error &http.httpError{err:"context deadline exceeded (Client.Timeout exceeded while reading body)", timeout:true} when reading response body. Please retry.
The underlying issue seems to be that there are roughly ~24K ConfigMaps for this particular Cluster and ~17K secrets. So obviously the return is going to be immense for both.
Is there any way to increase the read timeout for the body? Is there a paging feature, or is there anyway to implement one if there isn't?