Need better story for multiple datacenter scenarios

[mfischer-zd]

Currently, Vault isn't really practicable to use in a multi-datacenter environment. You can set up a Vault server and its backup server in a single datacenter, but there are at least two significant issues with this:

1. If the datacenter in which Vault resides experiences a network partition, it will be unreachable by clients in foreign datacenters.
2. If clients in foreign datacenters need to make many Vault requests serially (as is often the case during, for example, a Chef run, or when launching an application and storing values in the environment), the network latency can add a significant amount of time to the process. (Imagine 100 secrets need to be queried across a 500ms RTT link; that would take at least 50 seconds to finish, not counting TLS handshakes.)

Another option is to have a set of independent Vault instances in multiple datacenters, but it then becomes the administrator's job to ensure that the data is consistent among them all -- and that won't be practicable until the often-requested enumeration feature is implemented.

Consul can be used to replicate key-value data from a master datacenter to other datacenters (via consul-replicate). Consul is also one of the supported storage backends for Vault. So it seems logical that -- at least for generic secret storage backed by Consul -- one should be able to direct write requests to a Vault instance located in and connected to the "master" datacenter, and direct read requests to a Vault instance located in and connected to the closest replica datacenter.

Consul and PKI backends could probably also benefit from multi-datacenter support as well.

[jefferai]

but it then becomes the administrator's job to ensure that the data is consistent among them all -- and that won't be practicable until the often-requested enumeration feature is implemented.

Even when that lands I think you're better off using the physical store's built-in capabilities (whatever they are). Not everything will be able to be enumerated, and even if so, you won't catch e.g. the master keys this way.

[sandstrom]

This would be very useful. Both (1) and (2) would be an issue for us as well.

Something similar to High Availability, which some backends currently support, would be great. Consul + replication and some sugar to support this in Vault would be great.

[cetex]

Our plan is to run a separate vault cluster (on the same nodes that run consul masters for that DC) in each datacenter, since in our environment each datacenter is on it's own in a availability-zone/region/continent and may not have very good connectivity back to our core. then have a "master vault" somewhere secure which has the "master copy" of all secrets. This vault cluster will usually be selaed. To deploy a new or update secrets in a datacenter we would unseal this "master vault", pull the secrets and all configurations needed for a destination datacenter and push it to the destination vault, then reseal the "master vault" again.

This "master vault" would also be the only place where we store the root token for each destination datacenter.

If it would be possible to list everything inside vault we could easily retrieve / list all secrets and policies for any vault and quite easily script deploys / copying of (selected) secrets to new datacenters and keep track of / delete what shouldn't be there anymore.

So, the procedure would be something like this:

Unseal master vault - (These keys would be stored in a safe, as well as one key per security-person, to unseal without access to the safe we just need to make sure we have at least number of people who agree on it being unsealed)

init new "slave-vault" -> store keys in master vault
read unseal-keys for slave-dc from master vault -> unseal all vault's in slave-vault cluster. (would also verify that the keys work as stored)
read root-token for slave-dc from master vault -> auth to slave-vault (would also verify that the token works)

read /slave-vault-dc-name -> policies for slave-vault (list of secrets, policies, app-id's and such to deploy?, some datacenters only have few services, others need more)

Read / list all secrets / policies currently in slave-vault -> store in script as "current"
read each generic secret listed in slave-vault-dc-name -> store in script as "new"
read each dynamic secret policy listed in slave-vault-dc-name -> store / append in script to "new"
.... ... maybe more needed?
Diff "current" and "new"
Remove stuff from slave-vault that's in "current" and not in "new".
Add new stuff to slave-vault that's not in "current".
done?
seal master-vault

[jefferai]

#674 may be of interest. The person there is using consul-replicate to sync data from a master DC to separate HA Vault instances in other DCs. A minor bug to work out, but other than that it seems to be working.

[afterwords]

Handling some requests to be fulfilled locally like one-time use credentials and transit (encrypting) would also minimize unnecessary requests back to the master. Anything that doesn't require a verified write could be cached until it can successfully be written to the master.

[jefferai]

At this point I doubt we will ever have a distributed, asynchronous, eventually-consistent-with-a-master approach as @HorseHay is describing. There are a lot of complexities with this approach, and one thing we value very, very highly with Vault (due to it handling security-sensitive information) is predictability. The information in the article that I linked to from the mailing list could be used to distribute the transit keys to replicas, allowing local clients to use the local machines for transit; this would work quite well since transit doesn't generate leases.

[sandstrom]

Perhaps something simple to begin with, e.g. local caching (if that is indeed simple, perhaps it's not) or forwarding (where the local node simply forward to master + act as a standby). Perhaps this is already available today, and in that case the improvement may simply be to polish the documentation.

[sitano]

Hi all,

We have a big Consul multi-datacenter clusters deployment and we are going to deploy a separate independent Vault clusters along each consul quorum in each datacenter.

Each Vault cluster will have an independent master key (thus unsealing tokens) and will store datacenter specific secrets being at the same time independent from all others clusters. That's why consul-replication discussed in #674 would not work (the data is encrypted with its own keys in each dc).

What we are looking into is having an option of setting up application level partial replication (sub-tree). Virtually, we would like to have application level master-master replication abstracted off any certain backend. We are going to organize our own private internal Vaults to be a sink of all others and to provide kind of backup and duplication (so the system will be more reliable at the face of outage or splits - we do cross dc operations). Maybe the Vaults cross dc will be paired or whatever, but we want to have a choice on how we organize HA here with redundancy.

Having read what internet/github have on the topic, I came up with the following idea. I would be glad to have your feedback on this.

Latest Vault introduced request forwarding feature for HA mode. I am offering using that feature (or at least derivative of that), to replicate write requests based on configurable filters to any other configured datacenters. It could do write through or write back, but it will allow:

a) flexible propagation of the data across the replication streams
b) valid leases
c) the read cache can be enabled

The main problem this scheme involves is consistency. But for our usage scenario, its not a problem. Usually, we do not do parallel writes to multiple datacenters (so no conflicts on write events w/o any order) and we expect all datacenters to be available while rare MANUAL writing. If here something more secure required, the write ops can be implemented in sync way, so returning OK, only after all sinks are written. This could also guarantee consistency if add write rollbacks on conflicts for whoever want it.

[jefferai]

Replication is a part of Vault Enterprise which solves this use-case, closing!

[supine]

@jefferai I'm not sure "you can buy our proprietary version" is an appropriate reason to close a feature request on an open source project?

I can understand why Hashicorp might decline to work on it or even merge any of the resulting community work, but if others want to collaborate on it why interrupt that?