Update the logic for performing upgrades between releases

[swcurran]

The following issues were identified in trying to upgrade a 0.8.0-rc0 deployment to 0.8.0. All the issues are relatively straight forward, so putting them all together into one issue.

Issue: How to force an upgrade:

• Decision is to have both --force-upgrade and --from-version

Issue: What if the from-version is not in the upgrade yaml? It would be nice to have it fallback to the one previous to the --from-version or in secure storage.

• Decision: Make the code smarter and have it find the first version equal to or greater than the from version (from args or storage) and apply all the upgrades including and to the current version. This becomes the literal from to use.
• Further -- if there is more than one identical update -- run it only once. See the example above -- all of the updates are to resave-ConnRec so it really only needs to be done once.

Issue: Once a version is in storage, cannot move to a newer version without running the upgrade command.

• Decision: Change to only stop if there is an upgrade required between versions -- and if not, update the storage version.

Issue: On a new deployment, there is no version put into storage.

• Decision: Add to the create storage the additional of the version record.

Issue: No version in storage in existing deployments.

• Decision: Recommend that the deployment have an upgrade --from-version... command.

Issue: No warning currently produced when version in storage is not found.

• Decision: Add the warning, with details about what to do.

Issue: Can we run an upgrade automatically when deploying a new version. Current behaviour is to abort the startup.

• Option: Auto-upgrade.
  • Risk: Multiple instances try to run the upgrade could cause unexpected side effects.
    • Same as the next problem -- must be run offline
    • Need a semaphore or equivalent if the upgrade must be run once and only once.
  • Risk: Some updates MAY require running the upgrade offline.
    • Flag in YML file to say MUST be run as an explicit upgrade.

Issue: The current expectation is that in the YML, the version for the change is the "from" version -- e.g. when upgrading from a designated release.

• Decision: The version associated with an upgrade command should be the version in when the upgrade should be run. E.g. Any version being upgraded to that version must run the upgrade steps. So:
  • If the entry is v0.8.0: Update_Connections -- that command should be run from every pre-0.8.0 version when upgrade to 0.8.0 or beyond. Further, if going from 0.7.5 to 0.9.0, they would run the 0.8.0 upgrade.

[swcurran]

@shaangill025 -- if you think any of these are "too big" for what is an urgent update, please feel free to propose we separate out the change into a separate issue to be done in a later release. We can decide as a group how urgent it is and how therefore we will handle it.

Thanks