Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

API seal operation causes a panic/crash if a single use token is used #3497

Closed
rpsiv opened this issue Oct 27, 2017 · 0 comments
Closed

API seal operation causes a panic/crash if a single use token is used #3497

rpsiv opened this issue Oct 27, 2017 · 0 comments
Milestone
0.9.0

Comments

@rpsiv
Copy link

rpsiv commented Oct 27, 2017

Environment:
Vault Docker Image 0.8.3 running in OpenShift 3.5. The vault storage is consul 0.9.2

SKIP_SETCAP and disable_mlock are set to true. I do not believe these are related to the issue but are required since by default IPC_LOCK security context in OpenShift is not allowed.

  • Vault Version: 0.8.3
  • Operating System/Architecture: OpenShift 3.5 / Vault 0.8.3 Docker image

Vault Config File:

{
  "disable_mlock":"true",
  "storage":{
    "consul"
      {
        "address":"x.x.x.x:8500",
        "tls_ca_file":"/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt",
        "advertise_addr":"https://x.x.x.x:8500",
        "scheme":"https",
        "tls_skip_verify":"true",
        "path":"morty/",
        "service":"morty",
        "service_tags":"vault",
        "token":"<consul-token>"
    }
  },
  "listener":[
    {
      "tcp":{
        "address":"0.0.0.0:8200",
        "tls_key_file":"/etc/vault/tls/tls.key",
        "tls_cert_file":"/etc/vault/tls/tls.crt",
        "tls_disable":0
        }
      }
    ],
  "cluster_name":"morty-vault-morty"
}

Startup Log Output:

2017/10/27 14:13:55.638841 [INFO ] core: seal configuration missing, not initialized

  | 2017/10/27 14:13:55.713550 [INFO ] core: seal configuration missing, not initialized
  | 2017/10/27 14:13:55.807217 [INFO ] core: seal configuration missing, not initialized
  | 2017/10/27 14:13:56.639894 [INFO ] core: security barrier not initialized
  | 2017/10/27 14:13:56.682977 [INFO ] core: security barrier initialized: shares=5 threshold=3
  | 2017/10/27 14:13:56.776062 [INFO ] core: post-unseal setup starting
  | 2017/10/27 14:13:56.832792 [INFO ] core: loaded wrapping token key
  | 2017/10/27 14:13:56.832818 [INFO ] core: successfully setup plugin catalog: plugin-directory=
  | 2017/10/27 14:13:57.094487 [INFO ] core: successfully mounted backend: type=kv path=secret/
  | 2017/10/27 14:13:57.094524 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
  | 2017/10/27 14:13:57.094678 [INFO ] core: successfully mounted backend: type=system path=sys/
  | 2017/10/27 14:13:57.201209 [INFO ] expiration: restoring leases
  | 2017/10/27 14:13:57.201350 [INFO ] rollback: starting rollback manager
  | 2017/10/27 14:13:57.220364 [INFO ] expiration: lease restore complete
  | 2017/10/27 14:13:57.266935 [INFO ] core: post-unseal setup complete
  | 2017/10/27 14:13:57.266965 [INFO ] core/startClusterListener: starting listener: listener_address=0.0.0.0:8201
  | 2017/10/27 14:13:57.267129 [INFO ] core/startClusterListener: serving cluster requests: cluster_listen_address=[::]:8201
  | 2017/10/27 14:13:57.326775 [INFO ] core: root token generated
  | 2017/10/27 14:13:57.326790 [INFO ] core: pre-seal teardown starting
  | 2017/10/27 14:13:57.326795 [INFO ] core: stopping cluster listeners
  | 2017/10/27 14:13:57.326800 [INFO ] core: shutting down forwarding rpc listeners
  | 2017/10/27 14:13:57.326827 [INFO ] core: forwarding rpc listeners stopped
  | 2017/10/27 14:13:57.767360 [INFO ] core: rpc listeners successfully shut down
  | 2017/10/27 14:13:57.767383 [INFO ] core: cluster listeners successfully shut down
  | 2017/10/27 14:13:57.767468 [INFO ] rollback: stopping rollback manager
  | 2017/10/27 14:13:57.767552 [INFO ] core: pre-seal teardown complete
  | 2017/10/27 14:13:58.802108 [INFO ] core: vault is unsealed
  | 2017/10/27 14:13:58.802181 [INFO ] core: entering standby mode
  | 2017/10/27 14:13:58.847999 [INFO ] core: acquired lock, enabling active operation
  | 2017/10/27 14:13:58.910090 [INFO ] core: post-unseal setup starting
  | 2017/10/27 14:13:58.912645 [INFO ] core: loaded wrapping token key
  | 2017/10/27 14:13:58.912670 [INFO ] core: successfully setup plugin catalog: plugin-directory=
  | 2017/10/27 14:13:58.916166 [INFO ] core: successfully mounted backend: type=kv path=secret/
  | 2017/10/27 14:13:58.916387 [INFO ] core: successfully mounted backend: type=system path=sys/
  | 2017/10/27 14:13:58.916423 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
  | 2017/10/27 14:13:58.927453 [INFO ] expiration: restoring leases
  | 2017/10/27 14:13:58.927526 [INFO ] rollback: starting rollback manager
  | 2017/10/27 14:13:58.929233 [INFO ] expiration: lease restore complete
  | 2017/10/27 14:13:58.932248 [INFO ] core: post-unseal setup complete
  | 2017/10/27 14:13:58.932378 [INFO ] core/startClusterListener: starting listener: listener_address=0.0.0.0:8201
  | 2017/10/27 14:13:58.932779 [INFO ] core/startClusterListener: serving cluster requests: cluster_listen_address=[::]:8201
  | 2017/10/27 14:14:00.889987 [INFO ] core: enabled credential backend: path=approle/ type=approle
  | 2017/10/27 14:14:49.885128 [WARN ] core: stopping active operation
  | 2017/10/27 14:14:49.905517 [INFO ] core: pre-seal teardown starting
  | 2017/10/27 14:14:49.905533 [INFO ] core: stopping cluster listeners
  | 2017/10/27 14:14:49.905538 [INFO ] core: shutting down forwarding rpc listeners
  | 2017/10/27 14:14:49.905562 [INFO ] core: forwarding rpc listeners stopped
  | 2017/10/27 14:14:49.950739 [INFO ] core: rpc listeners successfully shut down
  | 2017/10/27 14:14:49.950753 [INFO ] core: cluster listeners successfully shut down
  | 2017/10/27 14:14:49.950778 [INFO ] rollback: stopping rollback manager
  | 2017/10/27 14:14:49.950866 [INFO ] core: pre-seal teardown complete
  | 2017/10/27 14:14:49.970748 [INFO ] core: vault is sealed
  | 2017/10/27 14:14:49 http: panic serving x.x.x.x:44642: runtime error: invalid memory address or nil pointer dereference
  | goroutine 568 [running]:
  | net/http.(*conn).serve.func1(0xc420406aa0)
  | /goroot/src/net/http/server.go:1697 +0xd0
  | panic(0x1ba1700, 0x2e25a10)
  | /goroot/src/runtime/panic.go:491 +0x283
  | github.com/hashicorp/vault/vault.(*TokenStore).Salt(0x0, 0x0, 0x0, 0x0)
  | /gopath/src/github.com/hashicorp/vault/vault/token_store.go:511 +0x41
  | github.com/hashicorp/vault/vault.(*TokenStore).SaltID(0x0, 0xc4207ebe90, 0x24, 0x2, 0x2, 0xbe74edb679dcfc22, 0x1010892cbf)
  | /gopath/src/github.com/hashicorp/vault/vault/token_store.go:631 +0x2f
  | github.com/hashicorp/vault/vault.(*TokenStore).Revoke(0x0, 0xc4207ebe90, 0x24, 0x0, 0x0)
  | /gopath/src/github.com/hashicorp/vault/vault/token_store.go:987 +0x127
  | github.com/hashicorp/vault/vault.(*Core).sealInitCommon.func1(0xc42052f970, 0xc4201da800, 0xc42052fa40, 0xc4207ebe90, 0x24)
  | /gopath/src/github.com/hashicorp/vault/vault/core.go:1141 +0x53
  | github.com/hashicorp/vault/vault.(*Core).sealInitCommon(0xc4201da800, 0xc420305340, 0x0, 0x0)
  | /gopath/src/github.com/hashicorp/vault/vault/core.go:1186 +0xa90
  | github.com/hashicorp/vault/vault.(*Core).SealWithRequest(0xc4201da800, 0xc420305340, 0x0, 0x0)
  | /gopath/src/github.com/hashicorp/vault/vault/core.go:1051 +0x171
  | github.com/hashicorp/vault/http.handleSysSeal.func1(0x2dcc380, 0xc420305260, 0xc420988800)
  | /gopath/src/github.com/hashicorp/vault/http/sys_seal.go:33 +0x113
  | net/http.HandlerFunc.ServeHTTP(0xc4203c8510, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:1918 +0x44
  | net/http.(*ServeMux).ServeHTTP(0xc420233860, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:2254 +0x130
  | github.com/hashicorp/vault/http.wrapHelpHandler.func1(0x2dcc380, 0xc420305260, 0xc420988800)
  | /gopath/src/github.com/hashicorp/vault/http/help.go:22 +0x166
  | net/http.HandlerFunc.ServeHTTP(0xc420274080, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:1918 +0x44
  | github.com/hashicorp/vault/http.wrapCORSHandler.func1(0x2dcc380, 0xc420305260, 0xc420988800)
  | /gopath/src/github.com/hashicorp/vault/http/cors.go:32 +0x10a
  | net/http.HandlerFunc.ServeHTTP(0xc4202740a0, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:1918 +0x44
  | github.com/hashicorp/vault/http.wrapGenericHandler.func1(0x2dcc380, 0xc420305260, 0xc420988800)
  | /gopath/src/github.com/hashicorp/vault/http/handler.go:88 +0xb1
  | net/http.HandlerFunc.ServeHTTP(0xc4202740e0, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:1918 +0x44
  | net/http.serverHandler.ServeHTTP(0xc420042680, 0x2dcc380, 0xc420305260, 0xc420988800)
  | /goroot/src/net/http/server.go:2619 +0xb4
  | net/http.(*conn).serve(0xc420406aa0, 0x2dcd780, 0xc4209837c0)
  | /goroot/src/net/http/server.go:1801 +0x71d
  | created by net/http.(*Server).Serve
  | /goroot/src/net/http/server.go:2720 +0x288

Expected Behavior:
Vault should have sealed and the API should have returned a 204 as documented.

Actual Behavior:
The log indicates the vault seals, however it immediately throws a panic
http: panic serving x.x.x.x:44642: runtime error: invalid memory address or nil pointer dereference
Steps to Reproduce:
Deploy the Vault
Initialize
Unseal
Created a single use root token (num_uses = 1)
PUT v1/sys/seal using that token

Important Factoids:
The issue appears to be related to the number of uses the token is allowed. When setting it to '2' or greater the /sys/seal endpoint returns a 204 as expected.

My suspicion is that the single use token is used up on the seal process and the a subsequent 'seal-status' is run to validate the seal completed. This would fail since the token is no longer valid. However a 5xx HTTP return code should be provided instead of a panic condition.

If that is true it would help to document how many token uses are required for the API endpoints. I have observed other issues related to that

References:
jefferai added a commit that referenced this issue Oct 27, 2017
@jefferai
Ensure revocation happens before seal/step-down since token store isn't
25ec3a0 
available after when using single-use tokens.

Fixes #3497
@jefferai jefferai added this to the 0.8.4 milestone Oct 27, 2017
@jefferai jefferai mentioned this issue Oct 27, 2017
Merged
jefferai added a commit that referenced this issue Nov 2, 2017
@jefferai
Ensure revocation happens before seal/step-down since token store isn…
66b2d26 
…'t (#3500)

available after when using single-use tokens.

Fixes #3497
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
0.9.0
Development

No branches or pull requests
2 participants
@jefferai @rpsiv