Introduce project manifest version 0.2.0 #495

stwiname · 2021-09-24T03:35:23Z

Addresses #463

Overview of changes:

Define new project manifest schema
Update validator for new schema, validate chain types
Update CLI for new schema
- ~~Update init to create a v0.0.2 project~~
- Update validate to validate all supported versions
Update Node for new schema
- Rely on genesis hash for network identification
- Add --network-registry flag to provide mapping from genesis hash to endpoint
- Support new chain types in external file
- Support new entry point for indexer sandbox

Todo:

~~Merge Update manifest to v0.0.2 subql-starter#30 and update init command once merged~~ Not merging so CLI releases before these changes are not broken
Setup and write tests for IndexerManager
~~Expand upon default values for --network-registry flag~~
Update docs (might be out of scope)

stwiname · 2021-09-24T03:36:39Z

packages/cli/src/commands/init.ts

-    project.endpoint = await cli.prompt('RPC endpoint', {
-      default: 'wss://polkadot.api.onfinality.io/public-ws',
+    project.network = await cli.prompt('Netork genesis hash', {
+      default: '0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3',


It might be worth adding more human readable options here. Like a list of network names and the ability to add a custom genesis

We can ask user for an endpoint and retrieve genesishash from that endpoint and ask confirm.

packages/cli/src/controller/init-controller.ts

packages/common/src/project/load.ts

ianhe8x · 2021-09-27T09:56:35Z

Probably need to reconsider the version, for this scale of change, i think we can use 0.2.0

ianhe8x · 2021-09-27T09:59:12Z

packages/common/src/project/types.ts

  specVersion: string;
  description: string;
  repository: string;
-  dataSources: SubqlDataSource[];
-  schema: string;


We can still have schema in IProjectManifest

I have removed it here because the type has changed from string to { file: string; }. Each of the versioned interfaces include the specific type.

I'm happy to add it back in as a union type or ensure that its mapped to { file: string; }

We can even make the type of schema GraphQLDocument or something alike.

packages/node/src/indexer/api.service.ts

packages/node/src/indexer/indexer.manager.ts

ianhe8x · 2021-09-27T10:17:32Z

@seandotau @zhex This is a quite large change for the project manifest. Your inputs are so important and welcomed.

stwiname · 2021-09-28T03:16:26Z

packages/node/src/configure/NodeConfig.ts

@@ -42,6 +46,14 @@ const DEFAULT_CONFIG = {
  indexCountLimit: 10,
  timestampField: true,
  proofOfIndex: false,
+  networkRegistry: {
+    /* TODO provide a default list */


This is probably the largest change on the hosted service side. If the project is using the new manifest spec then it only provides a genesis hash, so there needs to be a mapping to the endpoints.

ianhe8x · 2021-09-29T02:17:24Z

If there are still uncertainty in networkRegistry, we can leave it in the future PR. Currently we allow user provides overrides for endpoint, which should be enough.

stwiname · 2021-09-29T19:46:40Z

If there are still uncertainty in networkRegistry, we can leave it in the future PR. Currently we allow user provides overrides for endpoint, which should be enough.

I've removed the networkRegistry flag. networkEndpoint is required for v0.2.0 projects now and will throw an error if its not provided

ianhe8x · 2021-09-29T22:12:30Z

docs/references/references.md

@@ -35,6 +35,10 @@ Options:
      --timestamp-field     Enable/disable created_at and updated_at in schema
                                                       [boolean] [default: true]
  -d, --network-dictionary  Specify the dictionary api for this network [string]
+      --proof-of-index      Enable/disable proof of index
+                                                      [boolean] [default: false]
+      --network-registry    JSON object mapping genesisHash of each network to


update here since it has been removed

ianhe8x · 2021-09-29T22:14:54Z

packages/cli/src/commands/init.ts

+
+    if (flags.specVersion === '0.2.0') {
+      (project as ProjectSpecV0_2_0).genesisHash = await cli.prompt('Network genesis hash', {
+        default: '0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3', // Polkadot


for v0.2.0 we can still ask user for the endpoint first. with the endpoint we can

retrieve genesis hash and ask user to confirm it or they input another one

store endpoint in runtime config (is it included in v0.2.0?)

ianhe8x · 2021-09-29T22:26:03Z

If there are still uncertainty in networkRegistry, we can leave it in the future PR. Currently we allow user provides overrides for endpoint, which should be enough.

I've removed the networkRegistry flag. networkEndpoint is required for v0.2.0 projects now and will throw an error if its not provided

Sorry I change the idea and forget to update the issue. it says runtimeConfig.yaml only provided via cli flags. But after second though I think we can have it in the project.yaml (or have its reference), and remove it when we convert project.yaml to deployment..

We can have this change in a separate PR.

- Define new project manifest schema - Update validator for new schema, validate chain types - Update CLI for new schema - Update `init` to create a v0.0.2 project - Update validate to validate all supported versions - Update Node for new schema - Rely on genesis hash for network identification - Add `--network-registry` flag to provide mapping from genesis hash to endpoint - Support new chain types in external file - Support new entry point for indexer sandbox

…comments

…vided for v0.2.0 projects

…is missing

…o resolve genesis hash from url

stwiname · 2021-10-03T23:48:34Z

I can't seem to figure out why these tests cause other tests to fail https://github.com/subquery/subql/pull/495/files#diff-07519fb2dbc5155256ed036d08aece637848c043d153094ca7dab73099e6f34dR168-R180

My only thought could be because of something due to the sandbox vm or the process being overworked trying to index even though the fetch service gets stopped.

@ianhe8x any ideas on this? At this stage I'm happy just to disable these tests

* Introduce project manifest version 0.0.2 - Define new project manifest schema - Update validator for new schema, validate chain types - Update CLI for new schema - Update `init` to create a v0.0.2 project - Update validate to validate all supported versions - Update Node for new schema - Rely on genesis hash for network identification - Add `--network-registry` flag to provide mapping from genesis hash to endpoint - Support new chain types in external file - Support new entry point for indexer sandbox * Fix up tests * Fix up prettier * Fix lint issue * Enable noImplicitAny ts setting for cli, common and validator * Update new manifest version from 0.0.2 to 0.2.0, address other minor comments * Add tests for index manager, reuse sandbox VM for same entry point * Revert CLI init changes * Update docs * Update init command to allow specifying spec version * Remove networkDictionary option and require networkEndpoint to be provided for v0.2.0 projects * Remove logging * Add validator tests to jest scrope, fix valitation when project.yaml is missing * Fix failing test * Allow specifying an optional endpoint in network config, update CLI to resolve genesis hash from url * Fix commit hook issue with prettier and partially staged files * Fix tests failing * Disable breaking tests * Remove name from datasource in v0.2.0 manifest spec

stwiname requested a review from ianhe8x September 24, 2021 03:35

stwiname commented Sep 24, 2021

View reviewed changes

packages/cli/src/controller/init-controller.ts Outdated Show resolved Hide resolved

stwiname mentioned this pull request Sep 27, 2021

Implement publish command to upload project to IPFS #486

Merged

ianhe8x reviewed Sep 27, 2021

View reviewed changes

packages/common/src/project/load.ts Outdated Show resolved Hide resolved

ianhe8x requested changes Sep 27, 2021

View reviewed changes

ianhe8x requested a review from zhex September 27, 2021 10:16

stwiname force-pushed the project-manifest-0.0.2 branch from 785770c to b3d5efc Compare September 27, 2021 20:22

stwiname commented Sep 28, 2021

View reviewed changes

stwiname force-pushed the project-manifest-0.0.2 branch from dabf3b4 to a3f1f54 Compare September 29, 2021 02:34

ianhe8x reviewed Sep 29, 2021

View reviewed changes

ianhe8x mentioned this pull request Oct 1, 2021

support project from js files #384

Closed

stwiname added 12 commits October 1, 2021 14:18

Fix up tests

06fd5a4

Fix up prettier

9e15631

Fix lint issue

869e6e8

Enable noImplicitAny ts setting for cli, common and validator

359902b

Update new manifest version from 0.0.2 to 0.2.0, address other minor …

2c838b8

…comments

Add tests for index manager, reuse sandbox VM for same entry point

570f2d9

Revert CLI init changes

20bf812

Update docs

d0a7d7a

Update init command to allow specifying spec version

9930aad

Remove networkDictionary option and require networkEndpoint to be pro…

c8c298f

…vided for v0.2.0 projects

Remove logging

325df95

stwiname added 5 commits October 1, 2021 14:18

Add validator tests to jest scrope, fix valitation when project.yaml …

0658caf

…is missing

Fix failing test

e533edd

Allow specifying an optional endpoint in network config, update CLI t…

aa2c815

…o resolve genesis hash from url

Fix commit hook issue with prettier and partially staged files

8d67969

Fix tests failing

671bbe7

stwiname added 2 commits October 4, 2021 15:25

Disable breaking tests

608a9ad

Remove name from datasource in v0.2.0 manifest spec

e3db9ef

stwiname force-pushed the project-manifest-0.0.2 branch from b464ddc to e3db9ef Compare October 4, 2021 02:34

ianhe8x approved these changes Oct 4, 2021

View reviewed changes

ianhe8x merged commit d74daad into main Oct 4, 2021

ianhe8x deleted the project-manifest-0.0.2 branch October 4, 2021 03:35

jiqiang90 mentioned this pull request Oct 15, 2021

Import types from type-definitions #533

Closed

ianhe8x changed the title ~~Introduce project manifest version 0.0.2~~ Introduce project manifest version 0.2.0 Oct 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce project manifest version 0.2.0 #495

Introduce project manifest version 0.2.0 #495

stwiname commented Sep 24, 2021 •

edited

Loading

stwiname Sep 24, 2021

ianhe8x Sep 27, 2021

ianhe8x commented Sep 27, 2021

ianhe8x Sep 27, 2021

stwiname Sep 27, 2021

ianhe8x Sep 28, 2021

ianhe8x commented Sep 27, 2021

stwiname Sep 28, 2021

ianhe8x commented Sep 29, 2021

stwiname commented Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x commented Sep 29, 2021

stwiname commented Oct 3, 2021

Introduce project manifest version 0.2.0 #495

Introduce project manifest version 0.2.0 #495

Conversation

stwiname commented Sep 24, 2021 • edited Loading

Overview of changes:

Todo:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianhe8x commented Sep 27, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianhe8x commented Sep 27, 2021

Choose a reason for hiding this comment

ianhe8x commented Sep 29, 2021

stwiname commented Sep 29, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ianhe8x commented Sep 29, 2021

stwiname commented Oct 3, 2021

stwiname commented Sep 24, 2021 •

edited

Loading