[feature] pluggable storage

[jozef-slezak]

I would like to ask for a new feature: pluggable storage. Similarly to what Vault already has:

• https://github.com/hashicorp/vault/tree/master/plugins/database
• https://github.com/hashicorp/vault/tree/master/physical

It would great to add an option to configure different storage than bolt DB for Nomad (external KV, S3 or even SQL) because:

Motivation:

• Bolt is outdated (github repository is archived and read-only github.com/boltdb/bolt otherwise we could at least import maintained fork https://github.com/etcd-io/bbolt)
• Nomad Clients we would avoid "corrupt state" problems [feature] gracefully handle corrupt startup state #1367 (we are facing this issue even in 0.8.* tests)
• we could save peers/jobs/groups/tasks to the DB that has backups...
• deployments with existing storages could take advantage (idea: reuse existing storage)

I have done a small analysis (if you would be interested in this feature we can drill down into details):

1. Changes on the Nomad Client could be based on wrapped State Store:
   2d40f59
   a2b15a5
2. Each Nomad Server would have it's own namespace/bucket in the storage to store information needed to recover from outages https://www.nomadproject.io/guides/operations/outage.html
3. Nomad Server Leader could write to the storage HCLs (job/group/task...)

In our projects, we are using SQL databases, Consul.
I could provide pull requests with KV stored:
a. https://github.com/dgraph-io/badger
b. in SQL table.
c. Consul
d. ...

[OneCricketeer]

Duplicates #5378 ?

[tgross]

@Cricket007 I think this is mostly asking about Nomad's own internal storage (the "state store" where it stores information about jobs and whatnot), and not storage for tasks to use. It seems unlikely to me that we'd want to support external storage for Nomad's state because it would complicate reasoning about correctness, and pluggable state stores would complicate the number of test paths we'd have to cover.

That being said, the boltdb dependency is pretty stale. I think the original complaint here about state store corruption isn't something we've seen in production any time recently. Consul also relies on the same library as well.

[mikenomitch]

This isn't something we plan on doing any time in the near future, so I'm going to close this out.

As Tim mentioned, this would increase our number of test paths by a lot and make reasoning about state in Nomad a lot harder. While this is something that is technically possible (see Vault) since it adds a lot of complexity, we don't plan on it.

Going to close this out and will reopen a new issue if this is ever something we consider in the future.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.