Server-side Validation of Kubernetes yaml

user_1405787 0
Viewed 471

I would like to do server-side validation of Kubernetes yaml files before applying them.

I know that in my Jenkins agent, I could use the following kubectl command for validating yaml files at the server-side but I am a bit concerned about access-control:

  • Kubernetes < v1.18: kubectl apply --server-dry-run -f ...
  • Kubernetes >= v1.18: kubectl apply --dry-run=server -f ...

The Kubernetes documentation says the following:

Authorization for dry-run and non-dry-run requests is identical. Thus, to make a dry-run request, the user must be authorized to make the non-dry-run request.

I don't want any Jenkins agents to have super powers over my EKS cluster. A bad actor could use my Jenkins agent maliciously and apply any manifests they wanted. Right now for security/stability/management reasons, creating Kubernetes objects is done by a different system not Jenkins.

I checked a few other options but I can see drawbacks:

  • Kubeval is not aware of any CRDs installed in the actual cluster.
  • Client Validation is not really end-to-end validation
  • I could develop a rest api that exposes a validation rest endpoint and hits Kubernetes APIs or runs a kubectl --run-dry under the hood. However, this requires more dev work than we have capacity for.

Do you have any ideas or are you aware of any validation tools that I could use in our CI system securely for the purpose of validating end-2-end Kubernetes yaml files?

kubectl kubernetes jenkins
3 Answers

I've been looking for this myself and did not find a sufficient tooling. However, there are few workarounds:

  • Deploy all objects to a temporary ci-job-id namespace in dev/stage clusters. They should be the same as a prod, but will not impose the security risks you mentioned. This gives an additional benefit - you can check if everything got created, all pods are running. It helps to catch issues like insufficient resource requests, missing images, misconfigured Service selectors, etc. Also it let's you add a smoke test on top.
  • Spin a small minikube with all the CRDs specifically for CI validations. This approach gives you less coverage, but it is much cheaper to maintain.
user_9236200

Unfortunately what I wanted to do initially is not feasible. Applying server-side validation on a manifest containing objects with dependencies (e.g a Namespace and a Deployment)is not possible. See this issue.

As a result, client side validation is the solution moving forward. Either it is with Kubeval or any other client-side mechanism. Kubeval won't validate CRDs but there is away of ignoring them so that the validation does not fail:

Currently kubeval relies on schemas generated from the Kubernetes API. This means it's not possible to validate resources using CRDs. Currently you need to pass a flag to ignore missing schemas, though this may change in a future major version.

user_14057870

client-side validation (with kubectl --dry-run) has little value because it is missing some trivial schema misconfigurations.

you will get better results if you will perform the validation with kubeval (as you mentioned, you can skip CRDs) or you can use kubeconform that has CRD support.

user_95951060
Related