fix undefined suiteContext2020 object when importing with jest

[lemoustachiste]

When consuming with jest test runner, I was getting the following error:

/Users/julien/work/cert-verifier-js/node_modules/@digitalbazaar/ed25519-signature-2020/lib/Ed25519Signature2020.js:1
    TypeError: Cannot read properties of undefined (reading 'constants')

      at Object.<anonymous> (node_modules/@digitalbazaar/ed25519-signature-2020/lib/Ed25519Signature2020.js:14:44)
          at Object.<anonymous> (/Users/julien/work/cert-verifier-js/node_modules/@digitalbazaar/ed25519-signature-2020/lib/main.js:1)

coming from this line:

const SUITE_CONTEXT_URL = suiteContext2020.constants.CONTEXT_URL;

There was strangely no issue with the 2018 one. I noticed that ed25519-signature-2020-context was being called twice, the first time the value was undefined, the second it was the correct value.

I am not sure what exactly was happening, but this seems to fix it without changing the API of the package.

[davidlehn]

This is probably some module/cjs/esm/esm.js mixing issue with the tooling. This single line difference is probably to blame although it looks like it should function the same:

https://github.com/digitalbazaar/ed25519-signature-2018-context/blob/master/js/index.js#L12
contexts.set(CONTEXT_URL, context);

https://github.com/digitalbazaar/ed25519-signature-2020-context/blob/master/js/index.js#L12
contexts.set(constants.CONTEXT_URL, context);

May be better to fix the issue over in the 2020 context package if the 2018 one works.

[davidlehn]

And if anyone wants to properly debug this to understand the failure and what a proper fix is... well that would be nice too. ;-)

[lemoustachiste]

This is probably some module/cjs/esm/esm.js mixing issue with the tooling

Correct, my hunch is that this is related to the way things are exported, and jest does not pick it up correctly. I am actually facing other issues with other packages where the exported classes are empty and the methods undefined. I spent a good deal of time getting Ed25519VerificationKey2020.js and base58-universal to work and I also have to target the Ed25519Signature2020 file directly to get something:

from '@digitalbazaar/ed25519-signature-2020/lib/Ed25519Signature2020';

I don't really understand how you expose things ESM or CJS so I don't feel confident spending too much time on it, plus it's a pattern across all the libraries so it would be a big change that I cannot impose :)

https://github.com/digitalbazaar/ed25519-signature-2018-context/blob/master/js/index.js#L12
contexts.set(CONTEXT_URL, context);

https://github.com/digitalbazaar/ed25519-signature-2020-context/blob/master/js/index.js#L12
contexts.set(constants.CONTEXT_URL, context);

I think I tried that but it didn't change anything. I will try again and confirm.
I think it's tied to the way main and index and the rest of the package work together.

For instance I had to change base58btc to something like this:

// base58 characters (Bitcoin alphabet)
function getAlphabet (): string {
  return '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz';
}

As otherwise I was also getting an error:

cannot access alphabet before initialization

Again these are all jest concerns (at least I think so, I haven't tested the final browser build yet, but it was working fine as a nodejs script).
I am not an expert at the internals of jest, I just know it does not support Modules natively (so you have to go through something like babel-jest or ts-jest to transform files), and that it uses jsdom to emulate the browser.

[lemoustachiste]

https://github.com/digitalbazaar/ed25519-signature-2018-context/blob/master/js/index.js#L12
contexts.set(CONTEXT_URL, context);
https://github.com/digitalbazaar/ed25519-signature-2020-context/blob/master/js/index.js#L12
contexts.set(constants.CONTEXT_URL, context);
I think I tried that but it didn't change anything. I will try again and confirm.

Ok I tried again and that does not work

[lemoustachiste]

So to get to a working verification of Ed25519 signed credential with Jest, I need to:

1. import the Verification and Signature libraries as follows:

import { Ed25519VerificationKey2020 } from '@digitalbazaar/ed25519-verification-key-2020/lib/Ed25519VerificationKey2020';
import { Ed25519Signature2020 as Ed25519VerificationSuite } from '@digitalbazaar/ed25519-signature-2020/lib/Ed25519Signature2020';

which means this PR is actually irrelevant now

2. Both modify Ed25519VerificationKey2020.js and Ed25519Signature2020.js to

import * as base58btc from '../../../@blockcerts/hashlink-verifier/src/base58-universal/main';

Where I am preventing some sort of hoisting issue by wrapping the alphabet property in a function:
https://github.com/blockchain-certificates/hashlink-verifier/blob/master/src/base58-universal/main.ts (it is your code that I adjusted as per digitalbazaar/hashlink#25).
Without this change, I am getting the following error: "Cannot access 'alphabet' before initialization", a more complete output being:

{
      "verified": false,
      "results": [
        {
          "proof": {
            "@context": [
              "https://www.w3.org/2018/credentials/v1",
              "https://w3id.org/security/suites/ed25519-2020/v1",
              "https://w3id.org/blockcerts/v3"
            ],
            "type": "Ed25519Signature2020",
            "created": "2022-05-02T16:36:22.933Z",
            "verificationMethod": "did:key:z6MkjHnntGvtLjwfAMHWTAXXGJHhVL3DPtaT9BHmyTjWpjqs#z6MkjHnntGvtLjwfAMHWTAXXGJHhVL3DPtaT9BHmyTjWpjqs",
            "proofPurpose": "assertionMethod",
            "proofValue": "zAvFt59599JweBZ4zPP6Ge8LhKgECtBvDRmjG5VQbgEkPCiyMcM9QAPanJgSCs6RRGcKu96qNpfmpe9eTygpFZP6"
          },
          "verified": false,
          "error": {
            "name": "ReferenceError",
            "message": "Cannot access 'alphabet' before initialization",
            "stack": "ReferenceError: Cannot access 'alphabet' before initialization\n    at Object.decode (/Users/julien/work/cert-verifier-js/node_modules/base58-universal/main.js:19:25)\n    at Ed25519Signature2020.verifySignature (/Users/julien/work/cert-verifier-js/node_modules/@digitalbazaar/ed25519-signature-2020/lib/Ed25519Signature2020.js:106:38)\n    at Ed25519Signature2020.verifyProof (/Users/julien/work/cert-verifier-js/node_modules/jsonld-signatures/lib/suites/LinkedDataSignature.js:180:35)\n    at processTicksAndRejections (node:internal/process/task_queues:96:5)\n    at /Users/julien/work/cert-verifier-js/node_modules/jsonld-signatures/lib/ProofSet.js:236:53\n    at async Promise.all (index 0)\n    at _verify (/Users/julien/work/cert-verifier-js/node_modules/jsonld-signatures/lib/ProofSet.js:224:3)\n    at ProofSet.verify (/Users/julien/work/cert-verifier-js/node_modules/jsonld-signatures/lib/ProofSet.js:152:23)\n    at Object.verify (/Users/julien/work/cert-verifier-js/node_modules/jsonld-signatures/lib/jsonld-signatures.js:114:18)\n    at Ed25519Signature2020.checkDocumentStatus (/Users/julien/work/cert-verifier-js/src/suites/Ed25519Signature2020.ts:149:32)"
          }
        }
      ],

I will try and see how it behaves without the changes in the browser to isolate if this is only an issue with Jest.

[dlongley]

We should also note that #17 should fix this (but will involve a major version bump).

[lemoustachiste]

ok I'm still trying to test the browser build I have a issue with a dependency of jsonld-signature which does not work with a rollup plugin so I haven't tested.
If you want I can test the branch in my context to give you feedback, but that will be later.

[lemoustachiste]

After quite some hacking I have finally got a browser build and it verifies my document as expected.

I'll try and see if there is a high level configuration for jest that could work.

[lemoustachiste]

@dlongley so with the convert-to-module branch it all works without having to interfere with the code. Well almost.

I had to take each (all digital bazaar libraries) package.json and add:

"main": "./lib/index.js",

To note, this was also required to be able to build the project afterwards with rollup.

Apparently jest is supposed to support exports, but I'm on a older version due to another issue with node polyfills: jestjs/jest#9771

Rollup is also supposed to pick it up: rollup/plugins#208 but I've had the following if not done:

(!) Unresolved dependencies
https://rollupjs.org/guide/en/#warning-treating-module-as-external-dependency
@digitalbazaar/ed25519-verification-key-2020 (imported by src/suites/Ed25519Signature2020.ts)
@digitalbazaar/ed25519-signature-2020 (imported by src/suites/Ed25519Signature2020.ts)
@digitalbazaar/did-method-key (imported by src/domain/did/useCases/resolveDidKeyDocument.ts)

[davidlehn]

Sorry for all the hassle with this code! In theory it should "just work" but apparently that is not at all the case at the moment.

Looking at jest release notes, one would think it should handle what is going on here. But I see there are some jest/node issues related to imports. Maybe the double import thing you initially mentioned is part of that? Maybe serial vs parallel testing would help there?

Good to know the module conversion branch helps. I had thought all the modern module handling tools would work with just "exports" without "main". I had hoped to not have to have both.

The context packages are all a bit mixed up at the moment. They all do a similar thing of transforming cjs to esm. But they were written a while back and only have a "module" field. So bundlers will use the esm code, but node always uses cjs version. It did appear they all were working anyway so I was pushing off fixing all context packages until later. They will probably all switch around to module code that generates cjs. Or maybe just drop cjs? Not sure.

So in theory we're working towards all module code for the stack. And hopefully tools catch up with the style we're using. I'm not sure what to do if some tools are not quite there. Suggestions welcome.

[lemoustachiste]

Tomorrow I'll see how I can update Jest and Rollup, but I'm expecting trouble on that front which is why I had settled with the older versions.
Let's see, the train never stops so might as well update.

[lemoustachiste]

It looks like the support has just landed for Typescript: microsoft/TypeScript#33079 (comment). The announcement has been made on the 11th of May: https://devblogs.microsoft.com/typescript/announcing-typescript-4-7-rc/#esm-nodejs
I am also running jest 28 which is supposed to support it, I have installed v4.7.1-rc of typescript and set the module property to node16 in tsconfig.

Initially I have faced this error:

Module '@digitalbazaar/did-method-key' cannot be imported using this construct. The specifier only resolves to an ES module, which cannot be imported synchronously. Use dynamic import instead. 

Not only with this package but also with Ed25519VerificationKey2020
Apparently from online research it means the export must be set as default (see https://stackoverflow.com/questions/39415661/what-does-resolves-to-a-non-module-entity-and-cannot-be-imported-using-this).

So modifying the lib/index.js files with export default [DidKeyDriver|Ed25519VerificationKey2020], I manage to suppress the error.

And while this resolved importing the first level dependency, it seems it does not resolve correctly its own dependencies (it chokes on Cannot read properties of undefined (reading 'suite') which I suppose comes from one of those: https://github.com/digitalbazaar/did-method-key/blob/convert-to-module/lib/DidKeyDriver.js#L22)).

LIVE NOTE: I have applied the fix to the child deps, and now I am facing the issue with Ed25519VerificationKey2020 again, so it's pretty inconsistent and not really reliable/predictable. I swear I've managed to get 1 verification go through...

And I haven't started looking at rollup yet.

All in all it does not look like there is a straightforward support yet. I could prepare a repro if you wanted to work with it, but I also have deadlines coming up.

In the typescript documentation: https://devblogs.microsoft.com/typescript/announcing-typescript-4-7-beta/#package-json-exports-imports-and-self-referencing they still seem to use main as fallback, so I don't know if it's much effort for you, but it would be a great gain for me, i think. But if you can't then I can fork the libraries to adjust to my needs.

[lemoustachiste]

FWIW, I have forked all the involved packages in my project and added the main entry point.
I have both jest and rollup working as expected.

I'm going to close this PR, but happy to keep the conversation rolling.