Proof of index part 2 #488

jiqiang90 · 2021-09-21T10:00:01Z

Closes #410

TODO:

Fix comment issues
Folk MMR repo, fix relate bugs and test
Test if we set mmr to nullable will not cause conflict in our current projects
Solve when adding large number of blocks to filebaseDb, JavaScript heap out of memory issue

jiqiang90 · 2021-09-21T23:22:36Z

packages/node/src/indexer/mmr.service.ts

+    if (poiBlock.mmrRoot === null) {
+      poiBlock.mmrRoot = mmrValue;
+      await poiBlock.save();
+    } else if (u8aToHex(poiBlock.mmrRoot) !== u8aToHex(mmrValue)) {


Need to check Uint8Array value here, without u8aTohex this report two value is not the same.

use u8aEq

ianhe8x · 2021-09-29T01:07:09Z

packages/node/src/configure/NodeConfig.ts

+  get mmrPath(): string {
+    return this._config.mmrPath
+      ? this._config.mmrPath
+      : path.join(__dirname, `../../../../mmrs/${this.subqueryName}.mmr`);


The default path better is related to cwd not the js file.

ianhe8x · 2021-09-29T01:08:57Z

packages/node/src/indexer/entities/Poi.entity.ts

@@ -48,7 +48,7 @@ export function PoiFactory(sequelize: Sequelize, schema: string): PoiRepo {
      },
      mmrRoot: {
        type: DataTypes.BLOB,
-        allowNull: false,


this might cause compatibility issue for user already have POI table initialized

ianhe8x · 2021-09-29T01:09:40Z

packages/node/src/indexer/indexer.manager.ts

@@ -148,6 +146,7 @@ export class IndexerManager {
    await this.ensureMetadata(this.subqueryState.dbSchema);
    if (this.nodeConfig.proofOfIndex) {
      await this.poiService.init(this.subqueryState.dbSchema);
+      await this.mmrService.init(this.subqueryState.dbSchema);


can run them together

ianhe8x · 2021-09-29T02:30:11Z

packages/node/src/indexer/mmr.service.ts

+
+const logger = getLogger('mmr');
+const DEFAULT_WORD_SIZE = 32;
+const DEFAULT_LEAF = Buffer.from('0x000000000000000000000000000000');


Suggested change

const DEFAULT_LEAF = Buffer.from('0x000000000000000000000000000000');

const DEFAULT_LEAF = Buffer.alloc(32);

ianhe8x · 2021-09-29T02:37:04Z

packages/node/src/indexer/mmr.service.ts

+        `Append Mmr failed, input data length should be ${DEFAULT_WORD_SIZE}`,
+      );
+    }
+    const blockLeafHeight = poiBlock.id - this.blockOffset - 1; //leaf index alway -1


I found the blockOffset you implemented is '0', shouldn't it be '1' ? if we made this change, do we still need -1 here?

I think offset to 0 is make more sense than 1.

poiBlock.id - this.blockOffset should give us the the total number of blocks in the mmr (aka, leafLength)

blockOffset also used in line 46,

this.mmrStartHeight = (await this.fileBasedMmr.getLeafLength()) + this.blockOffset;

When mmr service initialization, we want to repeat calculate previous mmr value for validation. Eg, last fileDb mmr block height is 100, we want it repeat generate mmr for block 100 again and compare to value in poi table.
Therefore in this case, it start from 100 + 0 (which is equal to initial start block -1) .

And If we changed the blockOffset to 1 (which is equal to initial start block), it start from 100 + 1 (offset) -1 (to previous block).

ianhe8x · 2021-09-29T02:45:39Z

packages/node/src/indexer/mmr.service.ts

+      if (poiBlocks.length !== 0) {
+        for (const block of poiBlocks) {
+          if (this.mmrStartHeight + 1 < block.id) {
+            const nextAppendHeight = this.mmrStartHeight + 1;


nextAppendHeight is only used once, can make it inline

ianhe8x · 2021-09-29T02:46:40Z

packages/node/src/indexer/mmr.service.ts

+  private metadataRepo: MetadataRepo;
+  private fileBasedMmr: MMR;
+  private poiRepo: PoiRepo;
+  private mmrStartHeight: number;


can u use a better name, because mmrStartHeight doesn't sound like it will be changed.

ianhe8x · 2021-09-29T02:47:41Z

packages/node/src/indexer/mmr.service.ts

+  async syncFileBaseFromPoi(): Promise<void> {
+    while (!this.isShutdown) {
+      if (!this.poiRepo || !this.fileBasedMmr) {
+        await delay(10);


is it actually possible?

ianhe8x · 2021-09-29T02:48:51Z

packages/node/src/vendor/merkle-mountain-range/digests.ts

@@ -0,0 +1,42 @@
+// Copyright 2020-2021 OnFinality Limited authors & contributors


we shouldn't claim copyright for this file I believe.

ianhe8x · 2021-09-29T02:50:09Z

packages/node/src/vendor/merkle-mountain-range/fileBasedDb.ts

@@ -0,0 +1,162 @@
+// Copyright 2020-2021 OnFinality Limited authors & contributors


we shouldn't claim copyright for this file I believe.
And why you adding the whole merkle-mountain-range project? we only need to make some changes in FileBasedDb I think.

ianhe8x · 2021-10-03T05:17:37Z

packages/node/src/configure/NodeConfig.ts

+  get mmrPath(): string {
+    return this._config.mmrPath
+      ? this._config.mmrPath
+      : path.join(process.cwd(), `../../mmrs/${this.subqueryName}.mmr`);


Suggested change

: path.join(process.cwd(), `../../mmrs/${this.subqueryName}.mmr`);

: path.join(process.cwd(), `.mmr/${this.subqueryName}.mmr`);

And make sure the folder exists, otherwise mkdir first

ianhe8x · 2021-10-06T03:09:14Z

packages/node/src/configure/NodeConfig.ts

@@ -126,6 +127,12 @@ export class NodeConfig implements IConfig {
    return this._config.proofOfIndex;
  }

+  get mmrPath(): string {
+    return this._config.mmrPath


Suggested change

return this._config.mmrPath

return this._config.mmrPath ?? `.mmr/${this.subqueryName}.mmr`

ianhe8x · 2021-10-06T03:10:43Z

packages/node/src/indexer/mmr.service.ts

+
+const logger = getLogger('mmr');
+const DEFAULT_WORD_SIZE = 32;
+const DEFAULT_LEAF = Buffer.alloc(32);


Suggested change

const DEFAULT_LEAF = Buffer.alloc(32);

const DEFAULT_LEAF = Buffer.alloc(DEFAULT_WORD_SIZE);

ianhe8x · 2021-10-06T03:13:09Z

packages/node/src/indexer/mmr.service.ts

+  ensureFileBasedMmr(projectMmrPath: string) {
+    let fileBasedDb: FileBasedDb;
+    if (fs.existsSync(projectMmrPath)) {
+      fileBasedDb = FileBasedDb.open(projectMmrPath);


the flag as+ already cover the scenario that the file exists. I think the additional scenario need to be tend to is when the folder (not the file) doesn't exists.

* Draft * Mmr service draft * tidy up * Fix * fix mmr with forked mmr registry (experimental) * update test for indexer.manager * Fix lint * Update mmr dependency * Update mmr dependency * change mmr dependency to @subql/x-merkle-mountain-range * remove mmr vendor, use forked mmr repo * Fix * change default leaf * Fly-by fix, remove unused import

jiqiang90 requested a review from ianhe8x September 21, 2021 10:01

jiqiang90 commented Sep 21, 2021

View reviewed changes

ianhe8x requested changes Sep 29, 2021

View reviewed changes

ianhe8x reviewed Oct 3, 2021

View reviewed changes

jiqiang90 force-pushed the proof-of-index-pt2 branch from 334a60c to a19b2f3 Compare October 5, 2021 07:56

ianhe8x reviewed Oct 6, 2021

View reviewed changes

jiqiang90 added 12 commits October 13, 2021 12:03

Draft

eba2dca

Mmr service draft

5c9763a

tidy up

de27f77

Fix

395f1ab

fix mmr with forked mmr registry (experimental)

d024139

update test for indexer.manager

10bfaad

Fix lint

50c1841

Update mmr dependency

56a168a

Update mmr dependency

d105bff

change mmr dependency to @subql/x-merkle-mountain-range

dd34134

remove mmr vendor, use forked mmr repo

e3a1921

Fix

f8192fd

jiqiang90 force-pushed the proof-of-index-pt2 branch from 2d5f950 to f8192fd Compare October 13, 2021 00:56

jiqiang90 added 2 commits October 13, 2021 19:13

change default leaf

ed49f67

Fly-by fix, remove unused import

8466122

ianhe8x approved these changes Oct 14, 2021

View reviewed changes

ianhe8x merged commit 6a8d04f into main Oct 14, 2021

ianhe8x deleted the proof-of-index-pt2 branch October 14, 2021 04:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Proof of index part 2 #488

Proof of index part 2 #488

jiqiang90 commented Sep 21, 2021 •

edited

Loading

jiqiang90 Sep 21, 2021

jiqiang90 Sep 23, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

jiqiang90 Oct 1, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Sep 29, 2021

ianhe8x Oct 3, 2021

ianhe8x Oct 3, 2021

ianhe8x Oct 6, 2021

ianhe8x Oct 6, 2021

ianhe8x Oct 6, 2021

	const DEFAULT_LEAF = Buffer.from('0x000000000000000000000000000000');
	const DEFAULT_LEAF = Buffer.alloc(32);

		@@ -0,0 +1,42 @@
		// Copyright 2020-2021 OnFinality Limited authors & contributors

		@@ -0,0 +1,162 @@
		// Copyright 2020-2021 OnFinality Limited authors & contributors

	: path.join(process.cwd(), `../../mmrs/${this.subqueryName}.mmr`);
	: path.join(process.cwd(), `.mmr/${this.subqueryName}.mmr`);

	return this._config.mmrPath
	return this._config.mmrPath ?? `.mmr/${this.subqueryName}.mmr`

	const DEFAULT_LEAF = Buffer.alloc(32);
	const DEFAULT_LEAF = Buffer.alloc(DEFAULT_WORD_SIZE);

Proof of index part 2 #488

Proof of index part 2 #488

Conversation

jiqiang90 commented Sep 21, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiqiang90 commented Sep 21, 2021 •

edited

Loading