I am trying to initialize Vault on Kubernetes deployed through Helm in HA mode with TLS and Consul as the backend on EKS.
Helm 3.0.2
Kubernetes 1.17
Vault 1.5.4
Vault Helm Chart Version 0.8.0
Consul 1.8.4
Consul Helm Chart Version 0.25.0
The cluster is deployed through eksctl on AWS EKS.
The error I am seeing
helm install -f values.yaml hashicorp/vault
kubectl exec -it vault-o -- vault operator init
Error initializing: Put "https://127.0.0.1:8200/v1/sys/init": dial tcp 127.0.0.1:8200: connect: connection refused
My values.yaml
global:
tlsDisable: false
injector:
metrics:
enabled: true
server:
extraSecretEnvironmentVars:
- envName: AWS_ACCESS_KEY_ID
secretName: eks-creds
secretKey: AWS_ACCESS_KEY_ID
- envName: AWS_SECRET_ACCESS_KEY
secretName: eks-creds
secretKey: AWS_SECRET_ACCESS_KEY
extraVolumes:
- type: secret
name: vault-tls
- type: secret
name: eks-creds
standalone:
enable: false
ha:
enable: true
config: |
ui = true
api_addr = "[::]:8200"
cluster_addr = "[::]:8201"
listener "tcp" {
tls_disable = 0
tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
tls_key_file = "/vault/userconfig/vault-tls/vault.key"
tls_client_ca_file = "/vault/userconfig/vault-tls/vault.ca"
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "consul" {
path = "vault"
address = "HOST_IP:8501"
}
disable_mlock = true
service_registration "kubernetes" {}
ui:
enabled: true
I have tried changing api_addr, cluster_addr, address, and cluster_address over to PID_IP, https://127.0.0.1:8200, vault.default.svc.cluster.local, https://0.0.0.0:8200 and even a few other variants. So many I really haven't kept track. All with the same issue. I exec into the container and check the VAULT_ADDR env var. It's always reflected as I set it in the config. Even updated the VAULT_ADDR in the container to the vault kubernetes service ClusterIP. So I doubt it is this.
Some outputs in the vault container
/ $ env | grep VAULT_ADDR
/ $ VAULT_ADDR=https://127.0.0.1:8200
/ $ env | grep VAULT_CACERT
/ $ export VAULT_CACERT="/vault/userconfig/vault-tls/vault.ca"
/ $ ls -lAh /vault/userconfig/vault-tls/
total 0
drwxr-sr-x 2 root vault 100 Oct 27 18:21 ..2020_10_27_18_21_40.431431606
lrwxrwxrwx 1 root root 31 Oct 27 18:21 ..data -> ..2020_10_27_18_21_40.431431606
lrwxrwxrwx 1 root root 15 Oct 27 18:21 vault.ca -> ..data/vault.ca
lrwxrwxrwx 1 root root 16 Oct 27 18:21 vault.crt -> ..data/vault.crt
lrwxrwxrwx 1 root root 16 Oct 27 18:21 vault.key -> ..data/vault.key
I've tried checking the open/listening ports inside the container
/ $ netstat -tnlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
/ $
Which seems strange. But how would any of my value overrides cause this? So I feel like this is somehow intentional with the way it's setup. But maybe not.
/ $ wget https://127.0.0.1:8200
Connecting to 127.0.0.1:8200 (127.0.0.1:8200)
wget: can't connect to remote host (127.0.0.1): Connection refused
And while kubectl port-forward vault-0 8200
curl -v https://localhost:8200
* Trying ::1:8200...
* Connected to localhost (::1) port 8200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8200
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8200
Here is some kubernetes output
kubectl get pods
vault-0 0/1 Running 0 4s
vault-1 0/1 Running 0 4s
vault-2 0/1 Running 0 4s
vault-agent-injector-7bd76b66dd-8gjf4 1/1 Running 0 4s
kubectl get service
vault ClusterIP 10.100.165.241 <none> 8200/TCP,8201/TCP 2m1s
vault-active ClusterIP 10.100.22.197 <none> 8200/TCP,8201/TCP 2m1s
vault-agent-injector-svc ClusterIP 10.100.93.216 <none> 443/TCP 2m1s
vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 2m1s
vault-standby ClusterIP 10.100.96.141 <none> 8200/TCP,8201/TCP 2m1s
vault-ui ClusterIP 10.100.219.196 <none> 8200/TCP 2m1s
kubectl logs vault-0
WARNING! Unable to read storage migration status.
2020-10-28T03:31:47.639Z [INFO] proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-28T03:31:47.640Z [WARN] storage.consul: appending trailing forward slash to path
2020-10-28T03:31:47.640Z [WARN] storage migration check error: error="Unexpected response code: 400"
kubectl describe pod vault-0
Name: vault-0
Namespace: default
Priority: 0
Node: ip-172-16-3-151.ec2.internal/172.16.3.151
Start Time: Tue, 27 Oct 2020 20:31:46 -0700
Labels: app.kubernetes.io/instance=vault
app.kubernetes.io/name=vault
component=server
controller-revision-hash=vault-5dd945bc6c
helm.sh/chart=vault-0.8.0
statefulset.kubernetes.io/pod-name=vault-0
Annotations: kubernetes.io/psp: eks.privileged
Status: Running
IP: 172.16.3.19
IPs:
IP: 172.16.3.19
Controlled By: StatefulSet/vault
Containers:
vault:
Container ID: docker://1c70a35dbb2c870d21616e825200bb766fc10ab8076cd70931305f56386891fb
Image: vault:1.5.4
Image ID: docker-pullable://vault@sha256:121c1eb16a474f5a4c1d92256184dae333ab7284f8c744d4e2754300f84f68f0
Ports: 8200/TCP, 8201/TCP, 8202/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/sh
-ec
Args:
cp /vault/config/extraconfig-from-values.hcl /tmp/storageconfig.hcl;
[ -n "${HOST_IP}" ] && sed -Ei "s|HOST_IP|${HOST_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${POD_IP}" ] && sed -Ei "s|POD_IP|${POD_IP?}|g" /tmp/storageconfig.hcl;
[ -n "${HOSTNAME}" ] && sed -Ei "s|HOSTNAME|${HOSTNAME?}|g" /tmp/storageconfig.hcl;
[ -n "${API_ADDR}" ] && sed -Ei "s|API_ADDR|${API_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${TRANSIT_ADDR}" ] && sed -Ei "s|TRANSIT_ADDR|${TRANSIT_ADDR?}|g" /tmp/storageconfig.hcl;
[ -n "${RAFT_ADDR}" ] && sed -Ei "s|RAFT_ADDR|${RAFT_ADDR?}|g" /tmp/storageconfig.hcl;
/usr/local/bin/docker-entrypoint.sh vault server -config=/tmp/storageconfig.hcl
State: Running
Started: Tue, 27 Oct 2020 20:31:47 -0700
Ready: False
Restart Count: 0
Readiness: exec [/bin/sh -ec vault status -tls-skip-verify] delay=5s timeout=3s period=5s #success=1 #failure=2
Environment:
HOST_IP: (v1:status.hostIP)
POD_IP: (v1:status.podIP)
VAULT_K8S_POD_NAME: vault-0 (v1:metadata.name)
VAULT_K8S_NAMESPACE: default (v1:metadata.namespace)
VAULT_ADDR: https://127.0.0.1:8200
VAULT_API_ADDR: https://$(POD_IP):8200
SKIP_CHOWN: true
SKIP_SETCAP: true
HOSTNAME: vault-0 (v1:metadata.name)
VAULT_CLUSTER_ADDR: https://$(HOSTNAME).vault-internal:8201
HOME: /home/vault
AWS_ACCESS_KEY_ID: <set to the key 'AWS_ACCESS_KEY_ID' in secret 'eks-creds'> Optional: false
AWS_SECRET_ACCESS_KEY: <set to the key 'AWS_SECRET_ACCESS_KEY' in secret 'eks-creds'> Optional: false
Mounts:
/home/vault from home (rw)
/var/run/secrets/kubernetes.io/serviceaccount from vault-token-kfmn5 (ro)
/vault/config from config (rw)
/vault/userconfig/eks-creds from userconfig-eks-creds (ro)
/vault/userconfig/vault-tls from userconfig-vault-tls (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: vault-config
Optional: false
userconfig-vault-tls:
Type: Secret (a volume populated by a Secret)
SecretName: vault-tls
Optional: false
userconfig-eks-creds:
Type: Secret (a volume populated by a Secret)
SecretName: eks-creds
Optional: false
home:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
vault-token-kfmn5:
Type: Secret (a volume populated by a Secret)
SecretName: vault-token-kfmn5
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m16s default-scheduler Successfully assigned default/vault-0 to ip-172-16-3-151.ec2.internal
Normal Pulled 2m15s kubelet, ip-172-16-3-151.ec2.internal Container image "vault:1.5.4" already present on machine
Normal Created 2m15s kubelet, ip-172-16-3-151.ec2.internal Created container vault
Normal Started 2m15s kubelet, ip-172-16-3-151.ec2.internal Started container vault
Warning Unhealthy 21s (x22 over 2m5s) kubelet, ip-172-16-3-151.ec2.internal Readiness probe failed: Error checking seal status: Get "https://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
I port forward to localhost and get a PR_END_OF_FILE_ERROR error in my browser.
And I always have the same error. I understand why vault operator init is complaining. But I don't understand why that port is not allowing connection to it in regards to how vault has been configured. I'm running out of documentation to read and scowering the net for others with a similiar setup haveing the same error. Most of the time when I find others with this issue. They just have to export the VAULT_ADDR and it resolves the issue. Any ideas?