Unable to unseal hashicorp vault after system reboot

user_5753978 0
Viewed 3676

I installed vault locally and started, unsealed, and initialized the vault and added some secrets. After rebooting, I am unable to use the keys to unseal the vault. The first two unseal keys are accepted without issue, but after submitting the third key, I get an error response:

Error unsealing: Error making API request.

URL: PUT https://127.0.0.1:28200/v1/sys/unseal
Code: 500. Errors:

* failed to decrypt encrypted stored keys: cipher: message authentication failed

Any ideas what is going on? I am running vault version 1.4.2. The command I am using is vault operator unseal. The server config is:

vault_server.hcl

listener "tcp" {
  address = "127.0.0.1:28200"
  tls_cert_file = "/etc/vault/certs/vault_cert.crt"
  tls_key_file = "/etc/vault/certs/vault_cert.key"
}
storage "file" {
  path = "/etc/vault/mnt/data"
}

api_addr = "https://127.0.0.1:28200" # my $VAULT_ADDR is https://127.0.0.1:28200
disable_mlock = true

The relevant log output:

Jun 12 21:26:24 lambda vault[1147]: 2020-06-12T21:26:24.537-0500 [DEBUG] core: unseal key supplied
Jun 12 21:26:24 lambda vault[1147]: 2020-06-12T21:26:24.537-0500 [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=920f7d80-fdcc-3bc3-149e-8b069ef23acb
Jun 12 21:26:38 lambda vault[1147]: 2020-06-12T21:26:38.069-0500 [DEBUG] core: unseal key supplied
Jun 12 21:26:38 lambda vault[1147]: 2020-06-12T21:26:38.069-0500 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=920f7d80-fdcc-3bc3-149e-8b069ef23acb
Jun 12 21:26:51 lambda vault[1147]: 2020-06-12T21:26:51.984-0500 [DEBUG] core: unseal key supplied

The most relevant issues I can find in web searches are for people who inadvertently corrupted their storage:

I'm not sure that applies here. I'm using filesystem storage, vault is the owner of everything in /etc/vault, and I can't tell that any data has been lost or corrupted.

hashicorp-vault
3 Answers

I had the same issue with freshly installed vault 1.4.2 in HA mode on GKE using their official vault-k8s helm chart. I deployed it on 2 environments. The first one was OK, but the second one was failing exactly the same way as you described when I tried to join the 2nd vault instance to the HA cluster. I simply deleted and re-installed it a few times and eventually it worked.

user_137751230

TLDR Summary: vault will always accept keys until it hits the minimum count so it can attempt to assemble/use the resulting unseal key. Accepting keys is not an indicator of validity.

The keys distributed by the vault server are actually "shards" or "shares" (exact terminology changes between documenting sources) that are generated by splitting/sealing the master key using Shamir's secret Sharing. Because the master key cannot be decrypted without accepting the minimum number of shards (defaults to 3, but could be configured to a different value), the vault server has no way of determining if a provided shard is a valid until that minimum is provided so it can attempt to 1. generate an unseal key from shards, and 2. use resulting key against the master key.

Hashicorp provides a decent overview of the process here: https://www.vaultproject.io/docs/concepts/seal

More information on shamir and the math behind it here: https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

Now the bad news:

The error implies you are using an incorrect set of keys to unlock your vault: the resulting unseal key is incorrect. Either they have been changed or they are for a different vault. (maybe bad cut-n-paste newline injection?). While some have suggested a reinstall, I don't think this will solve anything. If you are desperate, you could try using a different version of vault in-case there is an unseal bug in one of your distros, but that's ... a reach.

user_133886020
user_62768640
Related