Support nested JSON objects throughout in API POST/PUT values

[jgiles]

Feature Request: Support nested JSON objects throughout in API POST/PUT values

Environment:

• Vault Version: Vault v0.9.1 ('87b6919dea55da61d7cd444b2442cabb8ede8ab1')
• Operating System/Architecture: OSX

Vault Config File:
Running -dev with default config.

Startup Log Output:

$ VAULT_DEV_ROOT_TOKEN_ID=test-root-token vault server -dev
==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
               Log Level:
                   Mlock: supported: false, enabled: false
        Redirect Address: http://127.0.0.1:8200
                 Storage: inmem
                 Version: Vault v0.9.1
             Version Sha: 87b6919dea55da61d7cd444b2442cabb8ede8ab1

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
Vault is configured to only have a single unseal key. The root
token has already been authenticated with the CLI, so you can
immediately begin using the Vault CLI.

The only step you need to take is to set the following
environment variables:

    export VAULT_ADDR='http://127.0.0.1:8200'

The unseal key and root token are reproduced below in case you
want to seal/unseal the Vault or play with authentication.

Unseal Key: jb7/sRFph/4l3OfeFzyirqq3xGwcGBGibokzbvXtyTo=
Root Token: test-root-token

==> Vault server started! Log data will stream in below:

2018/01/15 09:25:17.773080 [INFO ] core: security barrier not initialized
2018/01/15 09:25:17.773308 [INFO ] core: security barrier initialized: shares=1 threshold=1
2018/01/15 09:25:17.773487 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.786857 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.786872 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.789758 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.790010 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.790362 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.790899 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.793150 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.793175 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.793410 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.794486 [INFO ] identity: entities restored
2018/01/15 09:25:17.794559 [INFO ] identity: groups restored
2018/01/15 09:25:17.794587 [INFO ] core: post-unseal setup complete
2018/01/15 09:25:17.794863 [INFO ] core: root token generated
2018/01/15 09:25:17.794868 [INFO ] core: pre-seal teardown starting
2018/01/15 09:25:17.794873 [INFO ] core: cluster listeners not running
2018/01/15 09:25:17.794884 [INFO ] rollback: stopping rollback manager
2018/01/15 09:25:17.794932 [INFO ] core: pre-seal teardown complete
2018/01/15 09:25:17.795071 [INFO ] core: vault is unsealed
2018/01/15 09:25:17.795094 [INFO ] core: post-unseal setup starting
2018/01/15 09:25:17.795150 [INFO ] core: loaded wrapping token key
2018/01/15 09:25:17.795154 [INFO ] core: successfully setup plugin catalog: plugin-directory=
2018/01/15 09:25:17.795381 [INFO ] core: successfully mounted backend: type=kv path=secret/
2018/01/15 09:25:17.795537 [INFO ] core: successfully mounted backend: type=system path=sys/
2018/01/15 09:25:17.795704 [INFO ] core: successfully mounted backend: type=identity path=identity/
2018/01/15 09:25:17.795716 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2018/01/15 09:25:17.796976 [INFO ] expiration: restoring leases
2018/01/15 09:25:17.797181 [INFO ] rollback: starting rollback manager
2018/01/15 09:25:17.797346 [INFO ] identity: entities restored
2018/01/15 09:25:17.797442 [INFO ] identity: groups restored
2018/01/15 09:25:17.797559 [INFO ] expiration: lease restore complete
2018/01/15 09:25:17.797618 [INFO ] core: post-unseal setup complete

Expected Behavior:

1. The vault REST API would support (indeed, encourage) nested JSON values in POST/PUT requests
2. As a RESTful interface, the resources retrieved via GET would generally match those stored via POST in format.

Actual Behavior:

1. In at least some cases, Vault insists on string input values (like { "string_key": "string_serialized_value"... }
2. However, in responses to GET requests, the clearer nested JSON structure is used.

Steps to Reproduce:

$ VAULT_DEV_ROOT_TOKEN_ID=test-root-token vault server -dev
<snip>

# Simple example: Okta user/group objects.
$ vault auth-enable okta
Successfully enabled 'okta' at 'okta'!

# A proper list is the natural way to encode the set of groups in JSON.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/auth/okta/users/jdoe -d '{"groups": ["jets", "sharks"] }'
{"errors":["1 error occurred:\n\n* Error converting input [jets sharks] for field groups: '' expected type 'string', got unconvertible type '[]interface {}'"]}

# Ok, so we really have to pass it as a comma-separated string.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/auth/okta/users/jdoe -d '{"groups": "jets,sharks" }'

# But if we read it back, we get a proper list?
$ curl -H X-Vault-Token:test-root-token http://localhost:8200/v1/auth/okta/users/jdoe
{"request_id":"a497a60a-2808-a5c9-8c92-4cfbab27397b","lease_id":"","renewable":false,"lease_duration":0,"data":{"groups":["jets","sharks"],"policies":[""]},"wrap_info":null,"warnings":null,"auth":null}

# What if we full-on JSON serialize the input value?
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/auth/okta/users/jdoe -d '{"groups": "[\"jets\", \"sharks\"]" }'

# Eek!
$ curl -H X-Vault-Token:test-root-token http://localhost:8200/v1/auth/okta/users/jdoe
{"request_id":"a789a22d-83c5-58c2-fbd4-43252e171cdf","lease_id":"","renewable":false,"lease_duration":0,"data":{"groups":["[\"jets\"","\"sharks\"]"],"policies":[""]},"wrap_info":null,"warnings":null,"auth":null}

# Another example: Policies. From "Codifying Vault Policies and Configuration" blog post.
# Post a policy as a JSON object.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/sys/policy/postgresql-readonly -d '{"policy":{"path":{"postgresql/creds/readonly":{"capabilities":["read"]}}}}'
{"errors":["1 error occurred:\n\n* Error converting input map[path:map[postgresql/creds/readonly:map[capabilities:[read]]]] for field policy: '' expected type 'string', got unconvertible type 'map[string]interface {}'"]}

# Using a JSON-escaped string works.
$ curl -H X-Vault-Token:test-root-token -X POST http://localhost:8200/v1/sys/policy/postgresql-readonly -d '{"policy":"{\"path\":{\"postgresql/creds/readonly\":{\"capabilities\":[\"read\"]}}}"}'

# This time when we read it back, it stays a string value.
$ curl -H X-Vault-Token:test-root-token http://localhost:8200/v1/sys/policy/postgresql-readonly
{"name":"postgresql-readonly","rules":"{\"path\":{\"postgresql/creds/readonly\":{\"capabilities\":[\"read\"]}}}","request_id":"2b381515-49e2-20ce-5953-c488161a594e","lease_id":"","renewable":false,"lease_duration":0,"data":{"name":"postgresql-readonly","rules":"{\"path\":{\"postgresql/creds/readonly\":{\"capabilities\":[\"read\"]}}}"},"wrap_info":null,"warnings":null,"auth":null}

In summary, it seems like the API has made inconsistent decisions about how to treat complex nested values. As an API consumer, I would like to consistently construct, read and write JSON across the board, without the need for serializing nested values into strings.

As an aside, determining the proper JSON to use for policies was non-trivial - the output of json2hcl --reverse on the example HCL policy input did not work (it added extra list layers, and caused errors like {"errors":["Failed to parse policy: 1 error occurred:\n\n* path \"path\": invalid key 'postgresql/creds/readonly' on line 1"]})

References:

Confusion over posting policies: #582

[vishalnayak]

@jgiles In the Okta example, the field groups is taking in a value of type string as documented here. We parse this string based on a comma and consider each as a group name. But the returned type on the read is a JSON list and hence the output of ["[\"jets\"","\"sharks\"]"] is understandable.

However, this was the earlier way of handling a list input over the API. There was a new type that was introduced at a later time which handles this better (its TypeCommaStringSlice if you are interested in looking at the code). We adopted an incremental and need based transition of this type. The field groups in the Okta example you mentioned seems like a good candidate to be transitioned over.

As for the input for policies, the parsing of the input is performed by the hcl library which requires a string input and it doesn't look like we can accept a JSON input for the same. Since a single string input is desired, escaping it is required and the output you see, seem to be the expected behavior.

[jefferai]

Expanding on what Vishal said, a lot of Vault backends were built up at different times by different people and with the core supporting different features. At times as we go through and enhance backends with bug fixes or features we'll update them to newer practices, or we will do so as a result of specific issues/requests. I'm going to close this as a meta-issue, though -- please open issues for specific enhancement requests and we can figure out when/what milestones they should go on.

[jgiles]

Thanks for the quick response and code pointers!

Since the Okta issue looks like a simple bug, I took a stab at fixing it in #3801.

I've opened #3802 for the policy issue.