Hashicorp Vault - Agent Injector - does it make sense?

user_2586159 0
Viewed 1567

I have basic and fundamental question regards to Hashicorp Vault. I would like to inject some secrets (password to database) from Vault to container with Spring application.

I have prepared a specific annotations for using Vault with Kubernetes and everything works fine, saving PASSWORD variable as environment allows me using in application.properties file.

template:
metadata:
  annotations:
    vault.hashicorp.com/agent-inject: "true"
    vault.hashicorp.com/role: "myapp-role"
    vault.hashicorp.com/agent-inject-secret-foo: "secret/creds"
    vault.hashicorp.com/agent-inject-template-foo: |
      {{`{{- with secret "secret/creds" -}}
      PASSWORD={{ .Data.passcode }}
      {{- end }}`}}
  labels:
    app.kubernetes.io/name: {{ $appName }}
    app.kubernetes.io/instance: {{ .Release.Name }}
spec:
  containers:
    - name: {{ $appName }}
      image: "{{ .Values.vvvv.image.repository }}:{{ .Values.vvvv.image.tag }}"
      command: ["/bin/bash", "-c","while read line; do export $line; done < /vault/secrets/foo; /usr/local/tomcat/bin/catalina.sh run"]
      volumeMounts:
        - name: application-properties
          mountPath: /usr/local/tomcat/lib/application.properties
          subPath: application.properties
      ports:
        - name: http
          containerPort: 8080
          protocol: TCP

the question is pretty simple, does it make sense? inject agent saves the file with PLAIN text password in the /vault path, so everybody can see this secret... another question, how can I rotate credentials for application? should I use specific controllers inside the spring application right?

hashicorp-vault kubernetes-helm kubernetes
1 Answers

I think it definitely makes sense as the purpose is to avoid hard coding credentials in the spec.

inject agent saves the file with PLAIN text password in the /vault path, so everybody can see this secret.

Even in bare metal servers or cloud instances, credentials are saved in plain text. In case of k8s, it is inside a container. In both scenarios, you would control who can access your instances or k8s pods. Only authorised personnel shall have the required access to pods in production cluster(s).

how can I rotate credentials for application

The vault agent injector runs a sidecar container along with your app container in the same pod. Its purpose is to look for any changes in the vault secret at regular intervals. If you do a kubectl describe po <pod-name> you'll find a sidecar container vault-agent running.

kubectl get po app-example-deployment-7c4b45cf8-4fkr7
NAME                                     READY   STATUS    RESTARTS   AGE
app-example-deployment-7c4b45cf8-4fkr7   2/2     Running   0          166m

kubectl describe pod app-example-deployment-7c4b45cf8-4fkr7 :

...
vault-agent:
    Container ID:  docker://b6f9df32ed903d684c972401f41e15a8f6b1bec62aa111bfd9c693159af1ff09
    Image:         vault:1.7.0
    Image ID:      docker-pullable://vault@sha256:635cf1c3f9b10fe03aad375f94cc61f63d74a189662165285a8bf1c189ea04b8
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
      -ec
    Args:
      echo ${VAULT_CONFIG?} | base64 -d > /home/vault/config.json && vault agent -config=/home/vault/config.json
    State:          Running
      Started:      Tue, 13 Apr 2021 15:40:10 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     500m
      memory:  128Mi
    Requests:
      cpu:     250m
      memory:  64Mi
    Environment:
      VAULT_LOG_LEVEL:   info
      VAULT_LOG_FORMAT:  standard
...

After successfully fetching the secret from vault during deployment:

kubectl exec -it app-example-deployment-7c4b45cf8-4fkr7 -c app -- cat /vault/secrets/db-creds
mongodb+srv://testUser:testPass@test-5xxxx.mongodb.net/testDb

If I change the kv secret in Vault with password set to, let's say "testPass2", I need not do anything as the vault-agent sidecar container will update it for me automatically.

kubectl exec -it app-example-deployment-7c4b45cf8-4fkr7 -c app -- cat /vault/secrets/db-creds
mongodb+srv://testUser:testPass2@test-5xxxx.mongodb.net/testDb

In the vault-agent sidecar container logs, you'll see something like.

kubectl logs app-example-deployment-7c4b45cf8-4fkr7 -c vault-agent --follow
2021-04-13T14:40:10.426Z [INFO]  sink.file: creating file sink
2021-04-13T14:40:10.426Z [INFO]  sink.file: file sink configured: path=/home/vault/.vault-token mode=-rw-r-----
==> Vault agent started! Log data will stream in below:

==> Vault agent configuration:

                     Cgo: disabled
               Log Level: info
                 Version: Vault v1.7.0
             Version Sha: 4e222b85c40a810b74400ee3c54449479e32bb9f

2021-04-13T14:40:10.426Z [INFO]  template.server: starting template server
[INFO] (runner) creating new runner (dry: false, once: false)
2021-04-13T14:40:10.427Z [INFO]  auth.handler: starting auth handler
2021-04-13T14:40:10.427Z [INFO]  auth.handler: authenticating
2021-04-13T14:40:10.427Z [INFO]  sink.server: starting sink server
[INFO] (runner) creating watcher
2021-04-13T14:40:10.437Z [INFO]  auth.handler: authentication successful, sending token to sinks
2021-04-13T14:40:10.437Z [INFO]  auth.handler: starting renewal process
2021-04-13T14:40:10.437Z [INFO]  template.server: template server received new token
[INFO] (runner) stopping
[INFO] (runner) creating new runner (dry: false, once: false)
[INFO] (runner) creating watcher
[INFO] (runner) starting
2021-04-13T14:40:10.437Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
2021-04-13T14:40:10.439Z [INFO]  auth.handler: renewed auth token
[INFO] (runner) rendered "(dynamic)" => "/vault/secrets/db-creds"
2021-04-13T15:23:43.315Z [INFO]  auth.handler: renewed auth token
[INFO] (runner) rendered "(dynamic)" => "/vault/secrets/db-creds"
2021-04-13T16:07:16.191Z [INFO]  auth.handler: renewed auth token
user_17223260
Related