File Backend: Lease read error when unsealing if database role-name begins with "_"

[chrised]

Environment:

• Vault Version: Vault v0.8.3 ('6b29fb2b7f70ed538ee2b3c057335d706b6d4e36')
• Operating System/Architecture: Alpine Linux 3.6.2 x86_64 (Kubernetes/Docker)

Vault Config File:

{
    "listener": {
        "tcp": {
            "address": "[::]:8200",
            "tls_cert_file": "/opt/certs/vault.pem",
            "tls_client_ca_file": "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt",
            "tls_disable": "0",
            "tls_key_file": "/opt/certs/vault.key",
            "tls_min_version": "tls12",
            "tls_require_and_verify_client_cert": false
        }
    },
    "plugin_directory": "/mnt/plugins",
    "storage": {
        "file": {
            "path": "/mnt/data",
            "redirect_addr": "https://vault:8200"
        }
    }
}

Startup Log Output:

==> Vault server configuration:

                     Cgo: disabled
         Cluster Address: https://vault:8201
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", tls: "enabled")
               Log Level: trace
                   Mlock: supported: true, enabled: true
        Redirect Address: https://vault:8200
                 Storage: file
                 Version: Vault v0.8.3
             Version Sha: 6b29fb2b7f70ed538ee2b3c057335d706b6d4e36

==> Vault server started! Log data will stream in below:
2017/10/19 12:57:46.658053 [TRACE] physical/cache: creating LRU cache: size=32768
2017/10/19 12:57:46.787835 [TRACE] cluster listener addresses synthesized: cluster_addresses=[[::]:8201]
2017/10/19 12:57:49.488813 [DEBUG] core: cannot unseal, not enough keys: keys=1 threshold=3 nonce=cb33bac4-a1b6-651f-c790-0e84e3a78664
2017/10/19 12:57:49.491463 [DEBUG] core: cannot unseal, not enough keys: keys=2 threshold=3 nonce=cb33bac4-a1b6-651f-c790-0e84e3a78664
2017/10/19 12:57:49.493959 [INFO ] core: vault is unsealed
2017/10/19 12:57:49.496034 [INFO ] core: post-unseal setup starting
2017/10/19 12:57:49.496108 [TRACE] core: clearing forwarding clients
2017/10/19 12:57:49.496135 [TRACE] core: done clearing forwarding clients
2017/10/19 12:57:49.497246 [INFO ] core: loaded wrapping token key
2017/10/19 12:57:49.497325 [INFO ] core: successfully setup plugin catalog: plugin-directory=/mnt/plugins
2017/10/19 12:57:49.498466 [INFO ] core: successfully mounted backend: type=kv path=secret/
2017/10/19 12:57:49.498724 [INFO ] core: successfully mounted backend: type=system path=sys/
2017/10/19 12:57:49.498810 [INFO ] core: successfully mounted backend: type=database path=database/
2017/10/19 12:57:49.498884 [INFO ] core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2017/10/19 12:57:49.503621 [INFO ] expiration: restoring leases
2017/10/19 12:57:49.503911 [DEBUG] expiration: collecting leases
2017/10/19 12:57:49.504168 [INFO ] rollback: starting rollback manager
2017/10/19 12:57:49.504791 [INFO ] core: post-unseal setup complete
2017/10/19 12:57:49.511574 [DEBUG] expiration: leases collected: num_existing=18
2017/10/19 12:57:49.515212 [ERROR] expiration: error restoring leases: error=failed to read lease entry: read /mnt/data/sys/expire/id/database/creds/_system: is a directory
2017/10/19 12:57:49.515284 [ERROR] expiration: shutting down
2017/10/19 12:57:49.515328 [DEBUG] core: marked as sealed
2017/10/19 12:57:49.515365 [TRACE] core: clearing forwarding clients
2017/10/19 12:57:49.515402 [TRACE] core: done clearing forwarding clients
2017/10/19 12:57:49.515430 [INFO ] core: pre-seal teardown starting
2017/10/19 12:57:49.515458 [INFO ] core: cluster listeners not running
2017/10/19 12:57:49.515498 [DEBUG] expiration: stop triggered
2017/10/19 12:57:49.515528 [DEBUG] expiration: finished stopping
2017/10/19 12:57:49.515570 [INFO ] rollback: stopping rollback manager
2017/10/19 12:57:49.518257 [INFO ] core: pre-seal teardown complete
2017/10/19 12:57:49.518336 [DEBUG] core: sealing barrier
2017/10/19 12:57:49.518365 [INFO ] core: vault is sealed

Expected Behavior:

Vault should have unsealed, successfully reading previous leases

Actual Behavior:

Vault failed to unseal with an error referencing a path to a directory matching a database role with the name "_system"

Steps to Reproduce:

• Configure Vault for operation, with a File backend.
• Create a role in a database with a name that begins with an underscore.
• Request some credentials.
• Restart Vault.
• Unseal.

Important Factoids:

The database in question uses a custom, in-house plugin to allow user/role management within ArangoDB.
_system is a reserved table in ArangoDB, which defines other databases and users.
Please take note of the fact that I haven't tested this with other plugins, but this behaviour appears to be core to Vault (with the plugin having no interaction with the backend)
Contents of the _system directory were as follows:

# ls -lh _system/
total 4
-rw-------    1 root     root         891 Oct 19 12:44 _d3916017-658c-29e3-3ee4-4214a765af4d

Removing the _system directory from sys/expire/id/database/creds/ allowed Vault to start cleanly, albeit trashing all previous leases for the _system role/database.

[chrised]

I don't really have an opinion on what would be the correct fix for this. Perhaps just refusing to create a role with a name that starts with an underscore would be sufficient.

I can work around the issue by special-casing the _system table or modifying the way a role maps to a database within the plugin for now though.

[jefferai]

Would be fixed by, but a bit separate, from #2198