Cannot use helm upgrade in a gitlab-runner due forbidden secret access rules

user_1208538 0
Viewed 7156

I want to use execute helm on a gitlab-runner on my kubernetes in gitlab pipelines.

My gitlab.ci.yaml:

# Deployment step
deploy:
  stage: deploy
  image: alpine/helm:latest
  script:
    - helm --namespace gitlab upgrade initial ./iot/
  tags:
    - k8s
    - dev

What i have done so far:

  1. Installed the gitlab-runner on my kubernetes with helm (https://docs.gitlab.com/runner/install/kubernetes.html)

My values.yaml:

image: gitlab/gitlab-runner:alpine-v11.6.0

imagePullPolicy: IfNotPresent

gitlabUrl: https://gitlab.com/

runnerRegistrationToken: "mytoken"

unregisterRunners: true

terminationGracePeriodSeconds: 3600

concurrent: 10

checkInterval: 30

## For RBAC support:
rbac:
  create: true
  ## Define specific rbac permissions.
  # resources: ["pods", "pods/exec", "secrets"] 
  # verbs: ["get", "list", "watch", "create", "patch", "delete"]
  ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs cluster-wide or only within namespace
  clusterWideAccess: false

metrics:
  enabled: true
## Configuration for the Pods that that the runner launches for each new job
##
runners:
  ## Default container image to use for builds when none is specified
  ##
  image: ubuntu:16.04

  locked: false
  tags: "k8s,dev"
  privileged: true

  namespace: gitlab
  pollTimeout: 180
  outputLimit: 4096

  cache: {}
  ## Build Container specific configuration
  ##
  builds: {}
    # cpuLimit: 200m memoryLimit: 256Mi cpuRequests: 100m memoryRequests: 128Mi
  ## Service Container specific configuration
  ##
  services: {}
    # cpuLimit: 200m memoryLimit: 256Mi cpuRequests: 100m memoryRequests: 128Mi
  ## Helper Container specific configuration
  ##
  helpers: {}
securityContext:
  fsGroup: 65533
  runAsUser: 100
## Configure resource requests and limits ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources: {}

affinity: {}

nodeSelector: {}

tolerations: []

envVars:
    name: RUNNER_EXECUTOR
    value: kubernetes
## list of hosts and IPs that will be injected into the pod's hosts file
hostAliases: []

podAnnotations: {}

podLabels: {}
  1. gitlab-runner is succesfully connected with gitlab.com

But i get the following message on gitlab when executing the deployment step:

 Error: UPGRADE FAILED: query: failed to query with labels: secrets is forbidden: User "system:serviceaccount:gitlab:default" cannot list resource "secrets" in API group "" in the namespace "gitlab"

I've checked my RBAC ClusterRules and they are all per default set to a wildcard on verbs and ressources but i have also tried to set the needed rights:

  resources: ["pods", "pods/exec", "secrets"] 
  verbs: ["get", "list", "watch", "create", "patch", "delete"]

Nothing worked :-( When i have done wrong?

gitlab gitlab-ci-runner kubernetes rbac
3 Answers

I hope I have found the decision for this problem. Try to create clusterrolebinding like this

kubectl create clusterrolebinding gitlab-cluster-admin --clusterrole=cluster-admin --group=system:serviceaccounts
user_135245320

I fixed the same issue by setting runners.serviceAccountName to a serviceAccount I created with all rights in the namespace.

user_32339920

Here's my step to address this issue:

First, create service account and custom role for your Gitlab Runner:

kubectl create sa sa-runner -n gitlab
kubectl create role sa-runner-role -n gitlab --verb=get,list,watch,create,delete,patch --resource=pods,secret,pods/exec
kubectl create rolebinding sa-runner-rolebinding -n gitlab --role=sa-runner-role --serviceaccount=gitlab:sa-runner
kubectl create clusterrole deploy-default --verb=get,list,watch,patch --resource=deployment
kubectl create clusterrolebinding deploy-default-binding --clusterrole=deploy-default --serviceaccount=gitlab:default

Then, add serviceAccountName: sa-runner to your values.yaml file

## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
serviceAccountName: "sa-runner"

One you done with it, don't forget to do upgrade in your Helm

user_51258170
Related