GKE cluster (k8s 1.21) with dataplane v2 can't kubectl exec/port-forward/logs etc

user_6435778 0
Viewed 2385

We have a cluster at GKE which spontaneously stopped supporting kubectl commands exec/port-forward/logs upon cluster upgrade (seemingly to 1.21). Troubleshooting with https://cloud.google.com/kubernetes-engine/docs/troubleshooting#kubect_commands_stops did not solve the issue:

  • there are no egress-blocking firewall rules
  • control-plane-to-nodes ssh ingress rule is in place
  • there are four project-wide ssh keys set, this does not seem like "too many", it is unclear if it would be safe to remove some (and which?)
  • there is no ssh-key metadata on the node vm's
  • there are no egress-blocking network policies in place in k8s

Another cluster in the same GCP project (1.20, no dataplane v2) is working fine. The firewall rules are the same. We really have no clue what the problem could be, we can't find anything in the logs.

Does anybody have the same issue, or have any idea how we could troubleshoot this further?

Example output

$ kubectl -v 5 exec podname -- echo 'hi'
I0319 10:09:14.318262    8314 gcp.go:122] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credenkubectl -v 10 logs mainsite-test-next-web-deployment-7678d4ddc7-d57x5tial-plugins
I0319 10:09:14.396857    8314 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:09:14.396902    8314 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:09:14.396943    8314 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0319 10:09:14.419116    8314 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:09:14.419128    8314 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:09:14.454275    8314 podcmd.go:88] Defaulting container name to web
I0319 10:09:44.539168    8314 helpers.go:219] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "error dialing backend: dial timeout, backstop",
  "code": 500
}]
Error from server: error dialing backend: dial timeout, backstop

$ kubectl -v 5 logs podname 
I0319 10:12:47.021736   11845 gcp.go:122] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
I0319 10:12:47.086484   11845 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:12:47.086501   11845 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:12:47.086528   11845 shortcut.go:89] Error loading discovery information: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
I0319 10:12:47.110462   11845 request.go:1372] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
I0319 10:12:47.110480   11845 cached_discovery.go:78] skipped caching discovery info due to the server is currently unable to handle the request
I0319 10:13:17.158303   11845 helpers.go:219] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://10.164.0.17:10250/containerLogs/deploy-name/pod-name/web\": dial timeout, backstop",
  "code": 500
}]
Error from server: Get "https://10.164.0.17:10250/containerLogs/deploy-name/pod-name/web": dial timeout, backstop
google-kubernetes-engine kubernetes
2 Answers

As you have a "timeout" error it seems that kubectl can't communicate with the cluster control plane.

Can you give a try executing:

gcloud container clusters get-credentials CLUSTER_NAME
--region=COMPUTE_REGION

After this the communication with the control plane should be restored.

user_148462270

In the end it turned out that it was the network policies that we had set up for the kube-system namespace. This used to work just fine, until it didn't. We completely forgot that we had them in the first place :see_no_evil: Turns out we were a little overzealous there. When we removed them, everything was fine.

user_64357780
Related