Upgrade script for credx to anoncreds wallet

[ianco]

Update the aca-py upgrade process to support a version upgrade to anoncreds.

This ticket describes the overall task to create a script to upgrade an aca-py wallet from askar format to askar-anoncreds (to use with anoncred-rs). The specific wallet records to be updated are identified in issues #2384, #2385, #2386, #2389. This ticket covers the single-wallet scenario (one database, one wallet), the ticket to add support for multi-tenancy (and any multi-database-per-wallet scenarios) is issue #2785.

There is an existing script to upgrade from an Indy SDK wallet to Askar, located in the repository https://github.com/hyperledger/aries-acapy-tools. There is also a document describing this upgrade process here: https://github.com/hyperledger/aries-cloudagent-python/blob/main/docs/deploying/IndySDKtoAskarMigration.md

For the new askar to askar-anoncreds, it is recommended to:

• in the aries-acapy-tools repository, move the indy-sdk-to-askar code, documentation etc into a sub-directory (indy-to-askar-wallet-upgrade)
• in the aries-acapy-tools repository, create a new folder ('askar-to-anoncreds-wallet-upgrade) for the new wallet upgrade script
• create a new document AskarToAnoncredsRSMigration.md to describe the new upgrade process

Since Indy SDK is deprecated (and most aca-py users have likely already upgraded to Askar) it is preferable to keep the scripts and code separate between the two upgrade scripts. (If someone is still using Indy SDK they will need to first upgrade to Askar, and then from Askar to Askar-anoncreds.) Any documentation should also be separate (i.e. two separate README.md files, not one file that covers both upgrade types).

Regarding aca-py versions, the upgrade script should be developed and tested against the latest aca-py version (users should upgrade to the latest aca-py version before doing the wallet upgrade). Occasionally aca-py releases introduce wallet schema changes (usually just additional meta-data added to records) and theoretically this shouldn't affect the upgrade process, however if any wallet changes are introduced this assumption needs to be reviewed.

The upgrade itself should follow the same prerequisites/assumptions/etc that the existing indy sdk to askar script follows:

• users must first upgrade postgres and python versions
• user must make sure aca-py us updated to latest version and is running an Askar wallet
• user must shut down aca-py instances and make sure no one is using the database
• user must back up the wallet database

Ideally, users should also:

• complete any in-progress connection/issue credential/present proof protocols (not sure if these will be impacted or not by the upgrade process)
• publish any outstanding revocations (again, not sure how these will be impacted by the upgrade)

The askar to askar-anoncreds script should handle the same scenarios as the indy to askar script, and should use the same parameters, with the exception of some of the multitenancy logic (covered in another ticket):

• database upgrade is sqlite to sqlite or postgres to postgres (not sqlite to postgres)
• database parameters are required for the upgrade script
• the initial strategy to implement is --strategy dbpw (database per wallet)

The difference between Askar and Askar-anoncreds wallet is the format of some records, so the upgrade process just needs to read these records (in Askar format) and then write them back (in Askar-Anoncreds) format. (I.e. we don't need to replicate a lot of the logic in the existing indy-to-askar upgrade dealing with creating profiles etc.) The record types to upgrade are:

• update IssuerRevRegRecords - see issue Upgrade step for migrating from IssuerRevRegRecord #2385
• created schemas and cred defs - see issue Upgrade step for migrating from old created schema and cred def records #2386
• remove IssuerRevRegRecord - see issue Remove IssuerRevRegRecord #2384
• update private keys for cred defs and revocation registries (and anything else managed directly by credx) - see issue Upgrade step for indy-credx wallet entries #2389

[WadeBarnes]

(Actually the upgrade is more similar to the indy sdk -> askar process so cancel this question I'll review how that is setup and then update this ticket as appropriate ...)

Given this is more of a migration process, is a migration tool needed to migrate the wallet and records?

So, recommendation is we have a specific version number that will be the "last" credx-supported version (credx-ver), and then the next version number that will be the "first" anoncreds version (anon-ver). So based on the version number we will make an assumption about the wallet format. @swcurran is this a valid assumption? Otherwise we'll need to do more extensive updates to the upgrade function to work with wallet type vs version number.

Do we need to make any assumptions? Given you're migrating the wallet type from --wallet-type askar to --wallet-type askar-anoncreds (the same as done for --wallet-type indy to --wallet-type askar), does that not provide you with a clear indication of which wallet format to expect without making any assumption based on version.

[swcurran]

I agree with what @WadeBarnes says — that we are being explicit about the transition we are running, independent of versions.

As I understand it, a controller has two sets of API endpoints available, a new set that will use aries-anoncreds, and the other uses the “old” endpoints. A controller can use the new API endpoints and it automagically uses the new data structure. A controller with existing data from using the old endpoints that wants to switch to ONLY use the new API endpoints must run the upgrade to convert all of the relevant records to the new data structure and from then on must only use the new endpoints with those records. Presumably, the most impacted data is the revocation data for credential types being issued by the controller.

Is that correct? And if so, I don’t think the version comes into play.

[WadeBarnes]

Also given the explicit transition, any endpoints that are incompatible should automatically be disabled; they should return a descriptive not supported error when called, and be disabled or not displayed at all in the swagger interface. IMO

[ianco]

Yes the upgrade is a separate tool I had forgotten about that. I need to review it but I'll update the main description in this issue.

[ianco]

Also given the explicit transition, any endpoints that are incompatible should automatically be disabled; they should return a descriptive not supported error when called, and be disabled or not displayed at all in the swagger interface. IMO

Yes we already do that, protocols are loaded based on the wallet type (askar vs askar-anoncreds) which only enables the appropriate endpoints.

[ianco]

As I understand it, a controller has two sets of API endpoints available, a new set that will use aries-anoncreds, and the other uses the “old” endpoints. A controller can use the new API endpoints and it automagically uses the new data structure. A controller with existing data from using the old endpoints that wants to switch to ONLY use the new API endpoints must run the upgrade to convert all of the relevant records to the new data structure and from then on must only use the new endpoints with those records. Presumably, the most impacted data is the revocation data for credential types being issued by the controller.

Is that correct? And if so, I don’t think the version comes into play.

No it's all or nothing, either askar or askar-anoncreds. Converting from askar to askar-anoncreds will convert all records in the wallet and it will be a one-way conversion.

[ianco]

... does that not provide you with a clear indication of which wallet format to expect without making any assumption based on version.

There are no version-specific changes since 0.8.something (I think), but depending on if we do make structure changes the conversion too may need to know. For now we'll assume that aca-py has been upgraded to the most recent (which will be required to use askar-anoncreds anyways) and if there are any wallet structure changes we'll need to evaluate to see if the conversion tool needs to know about it.

[ianco]

FYI I've updated the main description to cover the upgrade details for a single wallet (no multitenancy, no multi-wallet-per-database), @swcurran @WadeBarnes @dbluhm can you take a look and see if I've missed anything? I have another ticket that I'll describe what's required for multi-whatevers.

[ianco]

Closing this issue, I'll create a new one