How to authenticate and access Kubernetes cluster for devops pipeline?

You should not copy your kubeconfig to the pipeline. Instead you can create a service account with permissions to a particular namespace and then use its credentials to access the cluster.

What I do is create a service account and role binding like this:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-tez-dev # account name
  namespace: tez-dev #namespace

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: tez-dev-full-access #role
  namespace: tez-dev
rules:
  - apiGroups: ["", "extensions", "apps"]
    resources: ["deployments", "replicasets", "pods", "services"] #resources to which permissions are granted
    verbs: ["*"] # what actions are allowed
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: tez-dev-view
  namespace: tez-dev
subjects:
  - kind: ServiceAccount
    name: gitlab-tez-dev
    namespace: tez-dev
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: tez-dev-full-access

Then you can get the token for the service account using:

kubectl describe secrets -n <namespace> gitlab-tez-dev-token-<value>

The output:

Name:         gitlab-tez-dev-token-lmlwj
Namespace:    tez-dev
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: gitlab-tez-dev
              kubernetes.io/service-account.uid: 5f0dae02-7b9c-11e9-a222-0a92bd3a916a

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1042 bytes
namespace:  7 bytes
token: <TOKEN>

In the above command, namespace is the namespace in which you created the account and the value is the unique value which you will see when you do

kubectl get secret -n <namespace>

Copy the token to your pipeline environment variables or configuration and then you can access it in the pipeline. For example, in gitlab I do (only the part that is relevant here):

k8s-deploy-stage:
  stage: deploy
  image: lwolf/kubectl_deployer:latest
  services:
    - docker:dind
  only:
    refs:
      - dev
  script:
     ######## CREATE THE KUBECFG ##########
    - kubectl config set-cluster ${K8S_CLUSTER_NAME} --server=${K8S_URL}
    - kubectl config set-credentials gitlab-tez-dev --token=${TOKEN}
    - kubectl config set-context tez-dev-context --cluster=${K8S_CLUSTER_NAME} --user=gitlab-tez-dev --namespace=tez-dev
    - kubectl config use-context tez-dev-context
    ####### NOW COMMANDS WILL BE EXECUTED AS THE SERVICE ACCOUNT #########
    - kubectl apply -f deployment.yml
    - kubectl apply -f service.yml
    - kubectl rollout status -f deployment.yml

The KUBECONFIG environment variable is a list of paths to Kubernetes configuration files that define one or more (switchable) contexts for kubectl (https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/).

Copy your Kubernetes configuration file to your pipeline agent (~/.kube/config by default) and optionally set the KUBECONFIG environment variable. If you got different contexts in your config file, you may want to remove the ones you don't need in your pipeline before copying it or switch contexts using kubectl config use-context.

Everything you need to connect to your kube api server is inside that config, certs, tokens etc.

If you don't want to copy a token into a file or want to use the API to automate the retrieval of the token, you can also execute some POST commands in order to programmatically retrieve your user token.

The full docs for this are here: https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_install#kube_api

The key piece is retrieving your id token with the POST https://iam.bluemix.net/identity/token call.

The body will return an id_token that you can use in your Kubernetes API calls.