Support non-serialized nested JSON policy objects

[jgiles]

Feature Request: Support non-serialized nested JSON policy objects

Environment:

• Vault Version: Vault v0.9.1 ('87b6919dea55da61d7cd444b2442cabb8ede8ab1')
• Operating System/Architecture: OSX

Vault Config File:
Running -dev with default config.

Startup Log Output:

$ VAULT_DEV_ROOT_TOKEN_ID=test-root-token vault server -dev
==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
               Log Level:
                   Mlock: supported: false, enabled: false
        Redirect Address: http://127.0.0.1:8200
                 Storage: inmem
                 Version: Vault v0.9.1
             Version Sha: 87b6919dea55da61d7cd444b2442cabb8ede8ab1

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.

The only step you need to take is to set the following
environment variables:

    export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.

Unseal Key: jb7/sRFph/4l3OfeFzyirqq3xGwcGBGibokzbvXtyTo=
Root Token: test-root-token

==> Vault server started! Log data will stream in below:

2018/01/15 09:25:17.773080 [INFO ] core: security barrier not initialized
2018/01/15 09:25:17.773308 [INFO ] core: security barrier initialized: shares=1 threshold=1
2018/01/15 09:25:17.773487 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.786857 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.786872 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.789758 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.790010 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.790362 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.790899 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.793150 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.793175 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.793410 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.794486 [INFO ] identity: entities restored
2018/01/15 09:25:17.794559 [INFO ] identity: groups restored
2018/01/15 09:25:17.794587 [INFO ] core: post-unseal setup complete
2018/01/15 09:25:17.794863 [INFO ] core: root token generated
2018/01/15 09:25:17.794868 [INFO ] core: pre-seal teardown starting
2018/01/15 09:25:17.794873 [INFO ] core: cluster listeners not running
2018/01/15 09:25:17.794884 [INFO ] rollback: stopping rollback manager
2018/01/15 09:25:17.794932 [INFO ] core: pre-seal teardown complete
2018/01/15 09:25:17.795071 [INFO ] core: vault is unsealed
2018/01/15 09:25:17.795094 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.795150 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.795154 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.795381 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.795537 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.795704 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.795716 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.796976 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.797181 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.797346 [INFO ] identity: entities restored
2018/01/15 09:25:17.797442 [INFO ] identity: groups restored
2018/01/15 09:25:17.797559 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.797618 [INFO ] core: post-unseal setup complete

Expected Behavior:

I expected to be able to POST policy objects as normal nested JSON.

Actual Behavior:

The server does not accept direct nested JSON policy objects; instead they must be passed as an escaped JSON string value.

Steps to Reproduce:

# Post a policy as a JSON object.
# Example policy from "Codifying Vault Policies and Configuration" blog post.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/sys/policy/postgresql-readonly -d '{"policy":{"path":{"postgresql/creds/readonly":{"capabilities":["read"]}}}}'
{"errors":["1 error occurred:\n\n* Error converting input map[path:map[postgresql/creds/readonly:map[capabilities:[read]]]] for field policy: '' expected type 'string', got unconvertible type 'map[string]interface {}'"]}

# Using a JSON-escaped string works.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/sys/policy/postgresql-readonly -d '{"policy":"{\"path\":{\"postgresql/creds/readonly\":{\"capabilities\":[\"read\"]}}}"}'

References:

#582 shows that this is not an unusual expectation.
Policy file from hashicorp/vault-provision-example shows how awkward things get when trying to write Vault policy resources for POSTing.
#3796 has some discussion of parsing HCL to JSON in the Vault client, which in this case would allow the whole file above to be written in HCL and used directly with vault write.

[jgiles]

I guess I should also state the converse: It would be nice to expose the policy values as straight JSON as well.

For example, I am writing CI tests for our checked-in policy + secret configuration. I am starting a local Vault server, provisioning all our configuration (with some fake root secrets like DB connection info), and then making assertions to sanity-check our configuration before it's merged and eventually deployed. For policies, a reasonable sanity-check is "all the secret paths referenced by all policies actually exist (i.e. there are no typos)". This would be very simple with JSON policies; you could write the test in any language with minimal dependencies. As is, I need to pull down the HCL string and parse it to extract the paths.

[jefferai]

We have no plans to change this, sorry.

[jgiles]

Is there a preferred process/forum for discussion of proposals like this? Has this been proposed and discarded before? A quick skim of the mailing list turned up mostly folks asking about problems sending policies via the API (which this might mitigate).

I definitely get keeping the GitHub issue list clean, but think this might merit more thought.

Please excuse the flurry of issues over the last few days as my team works through Vault integration. We've been thoroughly impressed with Vault so far - clean design, solid implementation. The few surprises/hiccups we've encountered (like this trickiness to posting policies via the API) would be great to smooth out while still in a 0.x.x release, and so we are raising them as we encounter them :-)

[jefferai]

Hi @jgiles,

Proposals like this can be discussed on the mailing list or on GH; unfortunately this particular proposal is simply something we can't satisfy. The sys/policies/acl endpoint may need to take in other parameters in the future, and having it accept pure JSON makes doing so impossible without more magic than I'm comfortable with (as a security product we try to keep magic out of Vault).

This isn't just theoretical; for EGPs we already accept additional parameters (https://www.vaultproject.io/api/system/policies.html#create-update-egp-policy). We separated out writing ACLs/RGPs/EGPs for cleanliness but had we stuck with sys/policy for all three we would have already run into this issue for ACLs too.

We realize that JSON-escaping a JSON-encoded string is annoying, but we need to leave the API some breathing room. One way to deal with this might be to use jq's tojson functionality which can take JSON and convert it to an escaped JSON string.

[jefferai]

BTW, thanks for the kind words :-D We know Vault has rough edges, thanks for bearing with them...