Why Nevermore?

If you imagine something like the variables table in Octopus Deploy, or the deployment process, you've got a lot of nested data structures:

• A project has a set of variables
• Each variable has a name
• The variable value can be of a specific type (sensitive, prompted at runtime, plain text, an x.509 certificate...)
• Variables can be scoped, e.g., here's the value for dev and test, but use that value for production

Storing this in a regular SQL table would involve a dozen tables or more. And every change to the model would require another data migration script. Loading up the variables page would mean a ton of joins, and saving it back would mean a lot of updates and change tracking.

When Octopus switched from RavenDB to SQL Server, we wanted to use SQL Server for storage, and wanted to make use of things like tables and columns and indexes. But we didn't want an explosion of tables for data that's unlikely to ever be queried directly.

Essentially, we wanted something like Dapper, but with a JSON blob at the end of each table for all of the complex nested data structures we needed to store, but didn't need to query much. This is where Nevermore came from.

Every now and then, we wonder. Does it make sense to maintain our own ORM? Why are we building and maintaining this thing? Should we use Dapper or EF instead?

You could look at it in a few ways:

Is storing our data as documents the right choice? For our scenario, with such complex documents, it seems to make sense. It worked really well when we used RavenDB, and it allows us to iterate more easily, and to add features to Octopus that if we weren't using documents we might decide were simply too difficult. So for us, documents make sense. If you start to imagine the data in Octopus without documents, you're looking at thousands of tables.

Is SQL Server the right choice then? For our customer base, the answer was historically "yes". For many customers, Octopus is a mission-critical system, and needs a HA story, and a backup story. They have people who know how to do that for SQL Server. They don't necessarily have people who know how to do that for a document database, nor for Postgres. This may change as we move outside the Microsoft ecosystem. For now, SQL Server is a known commodity.

So Nevermore is, admittedly, a compromise. If you don't need documents, you should use EF or Dapper. If you do need documents, maybe you shouldn't use SQL Server. If you find very compelling reasons to use both, you might need Nevermore.