Model versioning

[sstone1]

Introduction

This is a feature proposal for introducing explicit model versioning into the Concerto modelling language. Today, it is only possible to implement model versioning by convention, by inserting a version number into the namespace of a model. This makes it hard to understand what model versions exist in a set of models and how those model versions depend on each other.

We believe in the use of semantic versioning for models, which would mean that any instance created with a 1.x.y version (for example, 1.0.0) model should be compatible and validate against the latest 1.x.x version (for example, 1.2.3) model.

namespace [email protected]

concept Person {
  o String name
}

A given model manager would be able to load multiple versions of a model, so it could also load a new major version of that model with an incompatible change:

namespace [email protected]

concept Person {
  o String name
  o Integer age 
}

Namespaces

Namespaces may be versioned, which implicitly versions all the declarations within the namespace. We will adopt the SemVer.org grammar and semantics for version numbers.

Version numbers are composed of:

1. Major number
2. Minor number
3. Patch number
4. Pre-release identifier

Version numbers are encoded within namespaces using the following scheme (see semver.org for more extensive documentation):

namespace [email protected]

For example:

namespace [email protected]

Stable vs Unstable Versions

Versions containing a pre-release identifier are "unstable".

The latest stable version is the highest semantic version number that is not a pre-release.

The latest unstable version is the highest semantic version number, including pre-releases.

Unversioned Namespaces and Compatibility

Namespaces that do not have a version number will cause an exception to be thrown when unversioned namespaces are loaded.

Unversioned namespaces will be considered after all versioned namespaces. I.e. a model manager containing:

org.acme
[email protected]
[email protected]

Must consider [email protected] to be the latest stable version.

A model manager containing:

org.acme
[email protected]

Must consider org.acme to be the latest stable version of the org.acme namespace.

Imports

A model may reference an explicit version of a namespace (stable, or unstable) - and import types from that namespace.

A model must only reference a single version of a namespace. (not implemented)

Note: this is not a technical constraint, but is intended to discourage people creating namespaces that rely on types coming from different versions of the same imported namespace.

If a version number is specified if must be explicit - i.e. version ranges are not supported, because we do not have a “lock file” concept.

Importing Multiple Types

To facilitate importing multiple types from a versioned namespace a new import syntax must be supported:

import { Person, Order, Employee } from [email protected]

Backwards Compatibility

An import that does not specify a version identifier uses the latest “stable” version of a namespace. not implemented

In strict mode, unversioned imports must throw an exception.

import [email protected] // explicit use of an "unstable" pre-release
import [email protected] // explicit stable version
import org.acme.Product // uses latest "stable" (i.e. excludes pre-releases) — only available when `strict` mode is false

Importing from URL

When importing types from a URL the version number used in the import must match the version number of the imported namespace. not implemented

import [email protected] from https://acme.org/models/product.cto // explicit, loaded from a URL
import [email protected] from https://acme.org/models/product.cto // ERROR, if the CTO file at the URL has the wrong version
import org.acme.Product from https://acme.org/models/product.cto // explicit, loaded from a URL, irrespective of version — only available if `strict` mode is false

Instances

JSON serialisation encodes the version of a namespace used for a type. Validation checks that the instance conforms to the version of the namespace.

On serialization an instance may optionally be updated to reference the $class to reference the latest compatible version. not implemented

An instance would be able to refer to a model version using the model version number that the application used to create that instance:

[
  {
    "$class": "[email protected]",
    "name": "Simon"
  },
  {
    "$class": "[email protected]",
    "name": "Simon",
    "age": 99
  }
]

Backwards Compatibility

When deserialising, if a $class does not have a version then the latest stable version of the namespace will be used to validate the instance. not implemented

In strict mode the deserialization must throw an exception if the $class does not have a version number.

Model Manager

Model Manager manages a set of (versioned) namespaces. A model manager may contain version N as well as version N+1 of a namespace, as well as pre-release versions of the namespace.

Migration

Migration functions may be registered to migrate instances from version N to version N+1 of a namespace. Details TBD.

API Changes

Not implemented

BaseModelManager:

getModel(ns) // which version?

// filter by semver range
getModelFiles
getModels
getNamespaces()
getAssetDeclarations
getTransactionDeclarations
getEventDeclarations
getParticipantDeclarations
getEnumDeclarations
getConceptDeclarations

ModelFile:

getVersion() // return the version number for the namespace, or null if not specified
getFullyQualifiedTypeName // encode the version in the type name

[dselman]

I've started working on this.

The first step is to extend the CTO grammar to support specifying a version number for a namespace. The least disruptive change is to add a new VersionedNamespace grammar rule and modify the Model grammar rule to use it

Eg.

ModelVersion 
  = major:DecimalIntegerLiteral '.' minor:DecimalIntegerLiteral '.' patch:DecimalIntegerLiteral {
    return `${decimalIntegerLiteralToString(major)}.${decimalIntegerLiteralToString(minor)}.${decimalIntegerLiteralToString(patch)}`
  }

Namespace
  = NamespaceToken __ ns:QualifiedName ! '@' __ {
  	return ns;
  }

VersionedNamespace
  = NamespaceToken __ ns:QualifiedName '@' version:ModelVersion __ {
  	return `${ns}@${version}`;
  }

NamespaceDeclaration
	= Namespace /
      VersionedNamespace

Model
  = version:Version? ns:NamespaceDeclaration imports:Imports? body:SourceElements? {
      const result = {
        $class: "concerto.metamodel.Model",
        namespace: ns,
        imports: optionalList(imports),
        declarations: optionalList(body)
      };
      if (version) {
        result.concertoVersion = version;
      }
      return result;
    }

Note that this approach uses the parser to bake the version number into the namespace, which has some advantages and disadvantages:

• PRO: Do not need to modify the metamodel
• PRO: All existing code which uses the namespace as a key to index models continues to work (i.e. we don't have to change the identifier for a model to be namespace + version which I fear would be very impactful.
• PRO: Do not need to modify the metamodel -> CTO text generation
• CON: have to parse the namespace of a model to extract the version number. We can add a utility function for this?
• CON: have to decide what we do if we have org.acme and [email protected] in the model manager. Which is the "latest"?
• PRO/CON: instance serialization should work as-is, but will look like:

  {
     "$class" : "[email protected]",
     "name" : "Dan"
  }
  

Note that we still need to update the grammar rules for import to support major version imports. The code that resolves a type from an import declaration will have to be updated to be version aware: https://github.com/accordproject/concerto/blob/master/packages/concerto-metamodel/lib/metamodelutil.js#L228 — looking for the latest version of a specified major version number.

We will have to decide what we do in the validation code with explicit version numbers. For example, we could use the latest compatible version of a model for validation.

We will have to decide what we do when we generate JSON. For example, we could use the latest compatible version number of a model when we serialise an instance to JSON.

[mttrbrts]

PRO: All existing code which uses the namespace as a key to index models continues to work (i.e. we don't have to change the identifier for a model to be namespace + version which I fear would be very impactful.

Can you say a little more about this? As I understand it, this is exactly the motivation for this issue. I want to discriminate between versions and to take advantage of this, I would expect to have to update client code. What's the blast radius within AP?

[mttrbrts]

In the CTO Grammar, are you excluding pre-release versions deliberately?

[sstone1]

CON: have to decide what we do if we have org.acme and [email protected] in the model manager. Which is the "latest"?

Do we really want to support this mixed unversioned & versioned model story? Can we push unversioned models into some kind of "deprecated compatibility mode" and drop it in a later major version of Concerto?

[dselman]

In the CTO Grammar, are you excluding pre-release versions deliberately?

No. I've updated the description to include pre-releases.

[dselman]

CON: have to decide what we do if we have org.acme and [email protected] in the model manager. Which is the "latest"?

Do we really want to support this mixed unversioned & versioned model story? Can we push unversioned models into some kind of "deprecated compatibility mode" and drop it in a later major version of Concerto?

Yes, I think we could print some sort of warning for these and add a "strict" flag to the model manager to reject unversioned models. I think unversioned models would have to be considered last - i.e. they are conceptually "below" version 0.0.0, but are considered "stable".