How to fix k8s namespace permissions in gitlab ci

user_4925158 0
Viewed 1113

As I'm playing around with K8s deployment and Gitlab CI my deployment got stuck with the state ContainerStarting.

To reset that, I deleted the K8s namespace using kubectl delete namespaces my-namespace.

Now my Gitlab runner shows me 

$ ensure_namespace
Checking namespace [MASKED]-docker-3
error: the server doesn't have a resource type "namespace"
error: You must be logged in to the server (Unauthorized)

I think that has something to do with RBAC and most likely Gitlab created that namespace with some arguments and permissions (but I don't know exactly when and how that happens), which are missing now because of my deletion.

Anybody got an idea on how to fix this issue?

gitlab-ci kubernetes
4 Answers

In my case I had to delete the namespace in Gitlab database, so gitlab would readd service account and namespace:

On the gitlab machine or task runner enter the PostgreSQL console:

gitlab-rails dbconsole -p

Then select the database:

\c gitlabhq_production

Next step is to find the namespace that was deleted:

SELECT id, namespace FROM clusters_kubernetes_namespaces;

Take the id of the namespace to delete it:

DELETE FROM clusters_kubernetes_namespaces WHERE id IN (6,7);

Now you can restart the pipeline and the namespace and service account will be readded.

user_11254820

Deleting the namespace manually caused the necessary secrets from Gitlab to get removed. It seems they get autocreated on the first ever deployment and it's impossible to repeat that process.

I had to create a new repo and push to it. Now everything works.

user_49251580

Another solution is removing the cluster from Gitlab (under operations/kubernetes in your repo) and re-adding it.

user_117084280

From GitLab 12.6 you can simply clear the cluster cache.

To clear the cache:

  1. Navigate to your project’s Operations > Kubernetes page, and select your cluster.
  2. Expand the Advanced settings section.
  3. Click Clear cluster cache.

This avoids losing secrets and potentially affecting other applications.

XCondE_276930
Related