Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to create new GCP roleset, Vault server crashes #3

Closed
emilymye opened this issue Apr 12, 2018 · 4 comments
Closed

Unable to create new GCP roleset, Vault server crashes #3

emilymye opened this issue Apr 12, 2018 · 4 comments

Comments

@emilymye
Copy link
Contributor

emilymye commented Apr 12, 2018 •
edited
Loading

From hashicorp/vault#4344:

Environment:

Vault Version: v0.10.0
Operating System/Architecture: Arch Linux, kernel 4.15.5

Vault Config File:

/*
 * Vault configuration. See: https://vaultproject.io/docs/config/
 */

backend "file" {
	path = "/var/lib/vault"
}

listener "tcp" {
	/*
	 * By default Vault listens on localhost only.
	 * Make sure to enable TLS support otherwise.
	 */
	tls_disable = 1
}

Startup Log Output:

Apr 12 11:47:00 reeko systemd[1]: Started Vault server.
Apr 12 11:47:00 reeko vault[988]: ==> Vault server configuration:
Apr 12 11:47:00 reeko vault[988]:                      Cgo: enabled
Apr 12 11:47:00 reeko vault[988]:               Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", tls: "disabled")
Apr 12 11:47:00 reeko vault[988]:                Log Level: info
Apr 12 11:47:00 reeko vault[988]:                    Mlock: supported: true, enabled: true
Apr 12 11:47:00 reeko vault[988]:                  Storage: file
Apr 12 11:47:00 reeko vault[988]:                  Version: Vault v0.10.0
Apr 12 11:47:00 reeko vault[988]: ==> Vault server started! Log data will stream in below:
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.460-0700 [INFO ] core: vault is unsealed
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.460-0700 [INFO ] core: post-unseal setup starting
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.460-0700 [INFO ] core: loaded wrapping token key
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.460-0700 [INFO ] core: successfully setup plugin catalog: plugin-directory=
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.461-0700 [INFO ] core: successfully mounted backend: type=kv path=secret/
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.461-0700 [INFO ] core: successfully mounted backend: type=system path=sys/
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.461-0700 [INFO ] core: successfully mounted backend: type=identity path=identity/
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.462-0700 [INFO ] core: successfully mounted backend: type=gcp path=gcp/
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.462-0700 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.464-0700 [INFO ] core: restoring leases
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.464-0700 [INFO ] rollback: starting rollback manager
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.464-0700 [INFO ] expiration: lease restore complete
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.465-0700 [INFO ] identity: entities restored
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.465-0700 [INFO ] identity: groups restored
Apr 12 11:50:01 reeko vault[988]: 2018-04-12T11:50:01.465-0700 [INFO ] core: post-unseal setup complete
Apr 12 11:51:01 reeko vault[988]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 12 11:51:01 reeko vault[988]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x5648d7a0d27b]
Apr 12 11:51:01 reeko vault[988]: goroutine 205 [running]:
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin.(*backend).serviceAccountPolicyRollback(0xc420076ba0, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0x5648d95ef9a0, 0xc4208f9d40, 0x0, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin/rollback.go:162 +0x18b
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin.(*backend).walRollback(0xc420076ba0, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0xc420746b50, 0xa, 0x5648d95ef9a0, 0xc4208f9d40, 0xc4208f9ce0, 0xc42056e640)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin/rollback.go:33 +0x1a8
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin.(*backend).(github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin.walRollback)-fm(0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0xc420746b50, 0xa, 0x5648d95ef9a0, 0xc4208f9d40, 0x0, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vendor/github.com/hashicorp/vault-plugin-secrets-gcp/plugin/backend.go:65 +0x7f
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/logical/framework.(*Backend).handleWALRollback(0xc420596a90, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0x28, 0xc4207c5e00, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/logical/framework/backend.go:461 +0x343
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/logical/framework.(*Backend).handleRollback(0xc420596a90, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0x65313336b8009ce6, 0x6330632d32633463, 0xc420066b30)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/logical/framework/backend.go:409 +0x58
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/logical/framework.(*Backend).HandleRequest(0xc420596a90, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0x0, 0x0, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/logical/framework/backend.go:171 +0x696
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vault.(*Router).routeCommon(0xc4205aaac0, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0x5648d7360800, 0x0, 0x5648d99c0000, 0x0, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vault/router.go:530 +0x7b1
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vault.(*Router).Route(0xc4205aaac0, 0x5648d9a34b80, 0xc4205bf9c0, 0xc4205059e0, 0xc42017c360, 0xc420620c30, 0x4)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vault/router.go:381 +0x50
Apr 12 11:51:01 reeko vault[988]: github.com/hashicorp/vault/vault.(*RollbackManager).attemptRollback(0xc42017c360, 0x5648d9a34b80, 0xc4205bf9c0, 0xc420620c30, 0x4, 0xc4208668e0, 0x0, 0x0)
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vault/rollback.go:171 +0x2ae
Apr 12 11:51:01 reeko vault[988]: created by github.com/hashicorp/vault/vault.(*RollbackManager).startRollback
Apr 12 11:51:01 reeko vault[988]:         /build/vault/src/src/github.com/hashicorp/vault/vault/rollback.go:146 +0x13f
Apr 12 11:51:01 reeko systemd[1]: vault.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 12 11:51:01 reeko systemd[1]: vault.service: Failed with result 'exit-code'.

Expected Behavior:

We are attempting to use the GPC secrets backend to manage keys for a nightly backup job that writes files to a cloud storage bucket.

Actual Behavior:

vault write gcp/roleset/test-roleset ... successfully creates a GCP service account, but eventually fails with a 400. A few seconds later, vault server crashes with the log above.

Error writing data to gcp/roleset/redis-backup-roleset: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/gcp/roleset/test-roleset
Code: 400. Errors:

* unable to set policy: googleapi: Error 400: A policy to update must be provided., required

Steps to Reproduce:

vault secrets enable gcp
vault write gcp/config credentials="@vault-creds.json"
vault write gcp/roleset/test-roleset \
  project="my-project" \
  secret_type="access_token" \
  [email protected] \
  token_scopes="https://www.googleapis.com/auth/devstorage.read_write"

the bindings file contains:

resource "buckets/my-storage-bucket" {
  roles = [
    "roles/storage.objectCreator"
  ]
}

The vault-creds.json file is a service account key file generated by Google, for a service account with a role containing:

iam.serviceAccountKeys.create
iam.serviceAccountKeys.delete
iam.serviceAccountKeys.get
iam.serviceAccountKeys.list
iam.serviceAccounts.create
iam.serviceAccounts.delete
iam.serviceAccounts.get
iam.serviceAccounts.list
iam.serviceAccounts.update
storage.buckets.getIamPolicy
storage.buckets.setIamPolicy
storage.objects.getIamPolicy
storage.objects.setIamPolicy
@emilymye
Copy link
Contributor Author

emilymye commented Apr 12, 2018 •
edited
Loading

I'll add the fix to make sure the rollback doesn't fail, but I'm not sure why the update is failing. I'll need to investigate a little more. @naveg, please try it with debug logs on?

@naveg
Copy link

naveg commented Apr 12, 2018

Not much else:

Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=sys/
Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=auth/approle/
Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=identity/
Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=gcp/
Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=cubbyhole/
Apr 12 13:45:46 reeko vault[24753]: 2018-04-12T13:45:46.735-0700 [DEBUG] rollback: attempting rollback: path=auth/token/
Apr 12 13:45:46 reeko vault[24753]: panic: runtime error: invalid memory address or nil pointer dereference
Apr 12 13:45:46 reeko vault[24753]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x5641553de27b]

followed by the same stack trace as above

@emilymye
Copy link
Contributor Author

I found the issue - will have a fix out shortly. Thanks for your patience!

This was referenced Apr 12, 2018
Merged
Merged
@zBart
Copy link

zBart commented Apr 19, 2018

We are having the same issue and unfortunately Vault now crashes ~30 seconds after unsealing the vault.

Is there a way to update the plugin? Or something we can do to stop Vault from trying to rollback the service account policy?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@naveg @emilymye @zBart