feat(store): add splice events #1354

dk1a · 2023-08-22T20:32:06Z

TODO:

changeset-bot · 2023-08-22T20:32:12Z

🦋 Changeset detected

Latest commit: 9a2382c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 28 packages

Name	Type
@latticexyz/store	Major
@latticexyz/world	Major
@latticexyz/common	Major
@latticexyz/protocol-parser	Major
@latticexyz/dev-tools	Major
@latticexyz/store-sync	Major
create-mud	Major
@latticexyz/cli	Major
@latticexyz/react	Major
@latticexyz/store-indexer	Major
@latticexyz/block-logs-stream	Major
@latticexyz/config	Major
@latticexyz/abi-ts	Major
@latticexyz/ecs-browser	Major
@latticexyz/gas-report	Major
@latticexyz/network	Major
@latticexyz/noise	Major
@latticexyz/phaserx	Major
@latticexyz/recs	Major
@latticexyz/schema-type	Major
@latticexyz/services	Major
@latticexyz/solecs	Major
solhint-config-mud	Major
solhint-plugin-mud	Major
@latticexyz/std-client	Major
@latticexyz/std-contracts	Major
@latticexyz/store-cache	Major
@latticexyz/utils	Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

dk1a · 2023-08-24T12:20:01Z

The on-chain part is ready
Curious if you've already thought about the sync @alvrs @holic
The way I'm currently doing recs sync is bad, I guess we'd need to either do a middle bytes layer, or a splice parser that figures out which fields to update (with partial updates being the tricky part)

alvrs · 2023-08-31T11:58:13Z

packages/store-sync/src/blockLogsToStorage.ts

+            data: log.args.data,
+            start: log.args.start,
+            deleteCount: log.args.deleteCount,


Is there a way we could perform the steps to combine the new bytes with an existing record in here? Or would that prevent some optimizations we can do downstream (like encode+splice+decode in a single db tx)?

alvrs · 2023-08-31T12:01:01Z

packages/store/gas-report.json

@@ -693,7 +693,7 @@
    "file": "test/StoreCoreGas.t.sol",
    "test": "testSetAndGetField",
    "name": "set static field (1 slot)",
-    "gasUsed": 33280
+    "gasUsed": 33581


+300 in gas; some increase is expected, is there anything we can do to further optimize this or is this as good as it gets?

I didn't intentionally spend time optimizing gas here, probably can do better when I get to StoreCore

alvrs · 2023-08-31T12:11:06Z

packages/store/src/StoreCore.sol

+    // Prepare data for the splice event
+    uint256 start;
+    unchecked {
+      // (safe because it's a few uint40 values, which can't overflow uint48)
+      start = valueSchema.staticDataLength() + 32;
+      for (uint8 i; i < dynamicSchemaIndex; i++) {
+        start += encodedLengths.atIndex(i);
+      }
+    }
+    uint256 deleteCount = oldFieldLength;
+    // Emit event to notify indexers
+    emit StoreCore.StoreSpliceRecord(tableId, key, uint48(start), uint40(deleteCount), data);


Is it worth generalizing this in an internal util function, since very similar logic is used for _setDynamicField, _pushToDynamicField, _popFromDynamicField and _setDynamicFieldItem? I feel like might declutter the code a bit, make it easier to audit and easier to write tests for this logic in isolation

I've been thinking about this for a while now. I think the only efficient way to declutter is generalize it in a renderer function, and render StoreCore. Using solidity functions means more gas because the optimizer won't inline any of it and we'll need to return many vars

Or we could not worry about gas I guess, it's probably less than 1000 (haven't tested specifics). Though a bunch of return vars may still be ugly compared to a renderer with conditionals in the relevant places

Unfortunately I think a renderer would make auditing harder (because auditors have less experience with codegen). I'd be curious about the gas implications of moving it to a helper function, if it's a significant increase i'm also fine with keeping it inline

@alvrs do you want declutter in this PR? it feels like it'd be better in a separate one for easier reverts and gas comparisons

can try to refactor in a separate PR

alvrs · 2023-09-15T21:32:28Z

packages/store/src/PackedCounter.sol

+// We use a specific, invalid packed counter (total length != sum of lengths) to represent "unchanged" for the purpose of events and off-chain indexing
+bytes32 constant UNCHANGED_PACKED_COUNTER = bytes32(uint256(1));
+


do we still need this? doesn't seem like it's used

we're not using it yet so feel free to trash (which I see you've already done), can add later

alvrs

lgtm, amazing work @dk1a @holic. Let's get this over the finish line. Only thing missing is a changeset.

holic · 2023-09-16T12:18:42Z

added follow up issue for splice tests: #1515

Co-authored-by: Kevin Ingersoll <[email protected]> Co-authored-by: alvrs <[email protected]>

refactor StoreCore

731cb6a

vercel bot deployed to Preview August 22, 2023 20:33 View deployment

dk1a added 4 commits August 23, 2023 12:39

fixes

43d3597

refactor StoreCoreTest

3fb3400

refactor StoreCoreDynamicTest

a0da704

build

d68179c

vercel bot deployed to Preview August 23, 2023 11:49 View deployment

update comments and storeEvents

5b2c843

vercel bot deployed to Preview August 23, 2023 12:16 View deployment

tableId to table

d6cbd1d

alvrs changed the title ~~WIP StoreSpliceRecord~~ feat(store): add StoreSpliceField event Aug 24, 2023

dk1a added 2 commits August 24, 2023 14:56

hacky recs sync

cdca045

build

e896029

vercel bot deployed to Preview August 24, 2023 12:10 View deployment

alvrs changed the title ~~feat(store): add StoreSpliceField event~~ feat(store): add StoreSpliceRecord event Aug 24, 2023

alvrs reviewed Aug 31, 2023

View reviewed changes

Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent

27bbfce

vercel bot deployed to Preview September 1, 2023 11:58 View deployment

start refactoring storage adapters to be schemaless

07f3828

vercel bot deployed to Preview September 1, 2023 17:45 View deployment

holic added 2 commits September 4, 2023 09:58

use getTableConfig helper

6d74774

update sqlite storage adapter

3863505

vercel bot deployed to Preview September 4, 2023 09:36 View deployment

update postgres adapter

8783b68

vercel bot deployed to Preview September 4, 2023 10:06 View deployment

Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent

41957d0

alvrs reviewed Sep 15, 2023

View reviewed changes

alvrs mentioned this pull request Sep 15, 2023

Use splice events for hooks to avoid the linear storage reads #1510

Closed

review changes

1efdfbd

vercel bot deployed to Preview September 15, 2023 22:01 View deployment

gas report

83831d0

vercel bot deployed to Preview September 15, 2023 22:04 View deployment

remove unused variable

65eb794

vercel bot deployed to Preview September 15, 2023 22:06 View deployment

alvrs previously approved these changes Sep 15, 2023

View reviewed changes

Create shy-sheep-wait.md

3e3060d

holic dismissed alvrs’s stale review via 3e3060d September 15, 2023 22:45

vercel bot deployed to Preview September 15, 2023 22:46 View deployment

Update shy-sheep-wait.md

eea61a2

vercel bot deployed to Preview September 15, 2023 22:47 View deployment

Create fast-zebras-promise.md

6ad6fa7

vercel bot deployed to Preview September 15, 2023 22:49 View deployment

Create rotten-cats-lay.md

596fb15

vercel bot deployed to Preview September 15, 2023 22:52 View deployment

holic added 2 commits September 16, 2023 05:14

Create cyan-hats-try.md

0086bf9

Update shy-sheep-wait.md

9a2382c

vercel bot deployed to Preview September 16, 2023 12:16 View deployment

holic merged commit 331dbfd into main Sep 16, 2023

holic deleted the dk1a/storespliceevent branch September 16, 2023 12:23

github-actions bot mentioned this pull request Sep 16, 2023

Version Packages (next) #1465

Merged

This was referenced Sep 16, 2023

rename schemaIndex to fieldIndex #1479

Closed

Add missing events to Store (for setting partial fields) and review existing events #1121

Closed

separate data into staticData, encodedLengths, dynamicData in Store getter functions #1518

Closed

alvrs added a commit to Boffee/mud that referenced this pull request Sep 16, 2023

feat(store,): add splice events (latticexyz#1354)

fa351cc

Co-authored-by: Kevin Ingersoll <[email protected]> Co-authored-by: alvrs <[email protected]>

holic mentioned this pull request Sep 18, 2023

Add new Store events to client and indexer networking code #1141

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(store): add splice events #1354

feat(store): add splice events #1354

dk1a commented Aug 22, 2023 •

edited by holic

Loading

changeset-bot bot commented Aug 22, 2023 •

edited

Loading

dk1a commented Aug 24, 2023

alvrs Aug 31, 2023

alvrs Aug 31, 2023

dk1a Aug 31, 2023

alvrs Aug 31, 2023

dk1a Aug 31, 2023

dk1a Aug 31, 2023

alvrs Aug 31, 2023

dk1a Sep 1, 2023

alvrs Sep 4, 2023

alvrs Sep 15, 2023

holic Sep 15, 2023 •

edited

Loading

alvrs left a comment

holic commented Sep 16, 2023

		// We use a specific, invalid packed counter (total length != sum of lengths) to represent "unchanged" for the purpose of events and off-chain indexing
		bytes32 constant UNCHANGED_PACKED_COUNTER = bytes32(uint256(1));

feat(store): add splice events #1354

feat(store): add splice events #1354

Conversation

dk1a commented Aug 22, 2023 • edited by holic Loading

changeset-bot bot commented Aug 22, 2023 • edited Loading

🦋 Changeset detected

dk1a commented Aug 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holic Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

alvrs left a comment

Choose a reason for hiding this comment

holic commented Sep 16, 2023

dk1a commented Aug 22, 2023 •

edited by holic

Loading

changeset-bot bot commented Aug 22, 2023 •

edited

Loading

holic Sep 15, 2023 •

edited

Loading