Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store): add splice events #1354

Merged
merged 83 commits into from
Sep 16, 2023
Merged
Changes from 74 commits
Commits
Show all changes
83 commits
Select commit Hold shift + click to select a range
731cb6a
refactor StoreCore
dk1a Aug 22, 2023
43d3597
fixes
dk1a Aug 23, 2023
3fb3400
refactor StoreCoreTest
dk1a Aug 23, 2023
a0da704
refactor StoreCoreDynamicTest
dk1a Aug 23, 2023
d68179c
build
dk1a Aug 23, 2023
5b2c843
update comments and storeEvents
dk1a Aug 23, 2023
d6cbd1d
tableId to table
dk1a Aug 23, 2023
cdca045
hacky recs sync
dk1a Aug 24, 2023
e896029
build
dk1a Aug 24, 2023
27bbfce
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 1, 2023
07f3828
start refactoring storage adapters to be schemaless
holic Sep 1, 2023
6d74774
use getTableConfig helper
holic Sep 4, 2023
3863505
update sqlite storage adapter
holic Sep 4, 2023
8783b68
update postgres adapter
holic Sep 4, 2023
41957d0
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 4, 2023
3a30ac8
blockStorageOperations$ -> storedBlockLogs$
holic Sep 4, 2023
c3366e8
update dev tools
holic Sep 4, 2023
68a4a1a
extend event with packed counter, hydrate from indexer records
holic Sep 5, 2023
9c0e889
move splicing to util
holic Sep 5, 2023
c13dddc
build
holic Sep 5, 2023
b54782e
add missing args to tests
holic Sep 5, 2023
498ca2d
split dynamic and static data on-chain
dk1a Sep 6, 2023
652a724
refactor dynamicDataLengths to encodedLengths
dk1a Sep 7, 2023
529e472
update events
dk1a Sep 7, 2023
32ac8d6
refactor store tests, fix push
dk1a Sep 8, 2023
157d9ab
partially refactor world
dk1a Sep 8, 2023
e7db2a1
fix keyswithvalue
dk1a Sep 10, 2023
eab25bc
build
dk1a Sep 10, 2023
8677b55
Merge branch 'main' into dk1a/storespliceevent
dk1a Sep 11, 2023
9891b8d
Merge branch 'main' into dk1a/storespliceevent
dk1a Sep 11, 2023
d5b8828
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 12, 2023
ac6bcfd
get recs sync working
holic Sep 12, 2023
1514a11
attempt to suppress codegen diffs
holic Sep 12, 2023
b8d55ab
remove old typechain stuff
holic Sep 12, 2023
4eb2c1e
update postgres sync
holic Sep 12, 2023
06d660c
rename sqlite helpers to match postgres
holic Sep 12, 2023
7b8d66e
update sqlite sync
holic Sep 12, 2023
31b8ef8
fix import
holic Sep 12, 2023
789f91d
update test data
holic Sep 12, 2023
6fd0b15
fix StoreCore test
holic Sep 12, 2023
c25c463
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 12, 2023
fd2782c
Merge branch 'main' into dk1a/storespliceevent
alvrs Sep 13, 2023
7830808
update test snapshots
holic Sep 13, 2023
799e8ae
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 13, 2023
fdea79e
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 14, 2023
510ef7d
generate test data
holic Sep 14, 2023
a1d0015
update tests
holic Sep 14, 2023
2ef4d4a
fix tests
holic Sep 14, 2023
1c56dba
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 14, 2023
057d8e1
remove dupe/renamed method
holic Sep 14, 2023
cbdbf92
remove exhaustive check for now
holic Sep 14, 2023
0960120
fix dev tools
holic Sep 14, 2023
9956dc4
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 14, 2023
2d849b3
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 14, 2023
dda958d
update e2e setRecord calls
holic Sep 14, 2023
a9b3864
update test data
holic Sep 14, 2023
81f8f1b
wip failing test
holic Sep 14, 2023
a359faf
generate test data
alvrs Sep 14, 2023
bf90a34
add test for correct hex encoding to packed counter
alvrs Sep 14, 2023
838cebe
fix test input
alvrs Sep 14, 2023
2902fb1
don't reverse twice
holic Sep 14, 2023
567573e
add encodeLengths test
holic Sep 14, 2023
46d28e8
add spliceHex helper
holic Sep 14, 2023
f02845b
move spliceHex/readHex into common
holic Sep 14, 2023
30961a4
SpliceStaticRecord->SpliceStaticData, SpliceDynamicRecord->SpliceDyna…
holic Sep 14, 2023
af2bea5
Merge remote-tracking branch 'origin/main' into dk1a/storespliceevent
holic Sep 14, 2023
6b59547
that snuck in again
holic Sep 14, 2023
5bf1d35
fix ts error
holic Sep 14, 2023
df76123
update test data
holic Sep 14, 2023
310f6d4
Merge branch 'main' into dk1a/storespliceevent
alvrs Sep 15, 2023
4a07239
fixes
alvrs Sep 15, 2023
14e785a
fixes
alvrs Sep 15, 2023
31d3a44
mo fixes
alvrs Sep 15, 2023
297b5aa
add gas tests for encoding
alvrs Sep 15, 2023
1efdfbd
review changes
alvrs Sep 15, 2023
83831d0
gas report
alvrs Sep 15, 2023
65eb794
remove unused variable
alvrs Sep 15, 2023
3e3060d
Create shy-sheep-wait.md
holic Sep 15, 2023
eea61a2
Update shy-sheep-wait.md
holic Sep 15, 2023
6ad6fa7
Create fast-zebras-promise.md
holic Sep 15, 2023
596fb15
Create rotten-cats-lay.md
holic Sep 15, 2023
0086bf9
Create cyan-hats-try.md
holic Sep 16, 2023
9a2382c
Update shy-sheep-wait.md
holic Sep 16, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions e2e/packages/client-vanilla/src/mud/setupNetwork.ts
Original file line number Diff line number Diff line change
@@ -33,7 +33,7 @@ export async function setupNetwork() {
walletClient: burnerWalletClient,
});

const { components, latestBlock$, blockStorageOperations$, waitForTransaction } = await syncToRecs({
const { components, latestBlock$, storedBlockLogs$, waitForTransaction } = await syncToRecs({
world,
config: mudConfig,
address: networkConfig.worldAddress as Hex,
@@ -74,7 +74,7 @@ export async function setupNetwork() {
walletClient: burnerWalletClient,
worldContract,
latestBlock$,
blockStorageOperations$,
storedBlockLogs$,
waitForTransaction,
};
}
26 changes: 21 additions & 5 deletions e2e/packages/contracts/src/codegen/tables/Multi.sol

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

12 changes: 11 additions & 1 deletion e2e/packages/contracts/src/codegen/tables/Number.sol

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

19 changes: 15 additions & 4 deletions e2e/packages/contracts/src/codegen/tables/NumberList.sol

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

26 changes: 21 additions & 5 deletions e2e/packages/contracts/src/codegen/tables/Vector.sol

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion e2e/packages/contracts/worlds.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"31337": {
"address": "0x5FbDB2315678afecb367f032d93F642f64180aa3"
"address": "0x3Aa5ebB10DC797CAC828524e59A333d0A371443c"
}
}
48 changes: 30 additions & 18 deletions e2e/packages/sync-test/data/encodeTestData.test.ts
Original file line number Diff line number Diff line change
@@ -3,24 +3,36 @@ import { encodeTestData } from "./encodeTestData";

describe("encodeTestData", () => {
it("should encode numbers", () => {
expect(encodeTestData({ Number: [{ key: { key: 42 }, value: { value: 1337 } }] })).toStrictEqual({
Number: [
{
key: ["0x000000000000000000000000000000000000000000000000000000000000002a"],
value: "0x00000539",
valueSchema: "0x0004010003000000000000000000000000000000000000000000000000000000",
},
],
});
expect(encodeTestData({ Number: [{ key: { key: 42 }, value: { value: 1337 } }] })).toMatchInlineSnapshot(`
{
"Number": [
{
"dynamicData": "0x",
"encodedLengths": "0x0000000000000000000000000000000000000000000000000000000000000000",
"fieldLayout": "0x0004010004000000000000000000000000000000000000000000000000000000",
"key": [
"0x000000000000000000000000000000000000000000000000000000000000002a",
],
"staticData": "0x00000539",
},
],
}
`);

expect(encodeTestData({ Vector: [{ key: { key: 1337 }, value: { x: 42, y: -69 } }] })).toStrictEqual({
Vector: [
{
key: ["0x0000000000000000000000000000000000000000000000000000000000000539"],
value: "0x0000002affffffbb",
valueSchema: "0x0008020023230000000000000000000000000000000000000000000000000000",
},
],
});
expect(encodeTestData({ Vector: [{ key: { key: 1337 }, value: { x: 42, y: -69 } }] })).toMatchInlineSnapshot(`
{
"Vector": [
{
"dynamicData": "0x",
"encodedLengths": "0x0000000000000000000000000000000000000000000000000000000000000000",
"fieldLayout": "0x0008020004040000000000000000000000000000000000000000000000000000",
"key": [
"0x0000000000000000000000000000000000000000000000000000000000000539",
],
"staticData": "0x0000002affffffbb",
},
],
}
`);
});
});
23 changes: 8 additions & 15 deletions e2e/packages/sync-test/data/encodeTestData.ts
Original file line number Diff line number Diff line change
@@ -1,31 +1,24 @@
import { mapObject } from "@latticexyz/utils";
import { encodeKey, encodeValueArgs, valueSchemaToFieldLayoutHex } from "@latticexyz/protocol-parser";
import { Data, EncodedData } from "./types";
import { encodeAbiParameters, encodePacked } from "viem";
import config from "../../contracts/mud.config";
import { abiTypesToSchema, schemaToHex } from "@latticexyz/protocol-parser";
import { StaticAbiType } from "@latticexyz/schema-type";

/**
* Turns the typed data into encoded data in the format expected by `world.setRecord`
*/
export function encodeTestData(testData: Data) {
return mapObject(testData, (records, table) => {
if (!records) return undefined;
const keyConfig = config.tables[table].keySchema;
const tableConfig = config.tables[table];
return records.map((record) => {
const encodedKey = Object.entries(record.key).map(([keyName, keyValue]) => {
const keyType = keyConfig[keyName as keyof typeof keyConfig] as StaticAbiType;
return encodeAbiParameters([{ type: keyType }], [keyValue]);
});

const encodedValue = encodePacked(Object.values(config.tables[table].valueSchema), Object.values(record.value));

const encodedValueSchema = schemaToHex(abiTypesToSchema(Object.values(config.tables[table].valueSchema)));
const key = encodeKey(tableConfig.keySchema, record.key);
const valueArgs = encodeValueArgs(tableConfig.valueSchema, record.value);
const fieldLayout = valueSchemaToFieldLayoutHex(tableConfig.valueSchema);

return {
key: encodedKey,
value: encodedValue,
valueSchema: encodedValueSchema,
key,
...valueArgs,
fieldLayout,
};
});
}) as EncodedData<typeof testData>;
2 changes: 1 addition & 1 deletion e2e/packages/sync-test/data/expectClientData.ts
Original file line number Diff line number Diff line change
@@ -11,7 +11,7 @@ export async function expectClientData(page: Page, data: Data) {
for (const [table, records] of Object.entries(data)) {
for (const record of records) {
const value = await readComponentValue(page, table, encodeEntity(config.tables[table].keySchema, record.key));
expect(value).toEqual(record.value);
expect(value).toMatchObject(record.value);
}
}
}
6 changes: 4 additions & 2 deletions e2e/packages/sync-test/data/setContractData.ts
Original file line number Diff line number Diff line change
@@ -16,8 +16,10 @@ export async function setContractData(page: Page, data: Data) {
// TODO: add support for multiple namespaces after https://github.com/latticexyz/mud/issues/994 is resolved
tableIdToHex("", table),
record.key,
record.value,
record.valueSchema,
record.staticData,
record.encodedLengths,
record.dynamicData,
record.fieldLayout,
]);

// Wait for transactions to be confirmed
2 changes: 1 addition & 1 deletion e2e/packages/sync-test/data/types.ts
Original file line number Diff line number Diff line change
@@ -20,5 +20,5 @@ export type Datum<Table extends keyof (typeof config)["tables"] = keyof (typeof
export type Data = { [Table in keyof (typeof config)["tables"]]?: Array<Datum<Table>> };

export type EncodedData<T extends Data = Data> = {
[Table in keyof T]: Array<{ key: Hex[]; value: Hex; valueSchema: Hex }>;
[Table in keyof T]: Array<{ key: Hex[]; staticData: Hex; encodedLengths: Hex; dynamicData: Hex; fieldLayout: Hex }>;
};
1 change: 0 additions & 1 deletion e2e/packages/sync-test/rpcSync.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import { afterEach, beforeEach, describe, test } from "vitest";
import type { ViteDevServer } from "vite";
import { Browser, Page } from "@playwright/test";
import { ExecaChildProcess } from "execa";
import { createAsyncErrorHandler } from "./asyncErrors";
import { deployContracts, startViteServer, startBrowserAndPage, openClientWithRootAccount } from "./setup";
import {
2 changes: 1 addition & 1 deletion e2e/packages/sync-test/setup/startBrowserAndPage.ts
Original file line number Diff line number Diff line change
@@ -14,7 +14,7 @@ export async function startBrowserAndPage(

// log uncaught errors in the browser page (browser and test consoles are separate)
page.on("pageerror", (err) => {
console.log(chalk.yellow("[browser page error]:"), err.message);
console.log(chalk.yellow("[browser page error]:"), err.message, err);
});

// log browser's console logs
1 change: 0 additions & 1 deletion e2e/packages/sync-test/vite.config.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
import { defineConfig } from "vitest/config";

// TODO should this along with `.test.ts` be in `client-vanilla`?
export default defineConfig({
test: {
environment: "jsdom",
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ export async function setupNetwork() {
onWrite: (write) => write$.next(write),
});

const { components, latestBlock$, blockStorageOperations$, waitForTransaction } = await syncToRecs({
const { components, latestBlock$, storedBlockLogs$, waitForTransaction } = await syncToRecs({
world,
config: mudConfig,
address: networkConfig.worldAddress as Hex,
@@ -75,7 +75,7 @@ export async function setupNetwork() {
publicClient,
walletClient: burnerWalletClient,
latestBlock$,
blockStorageOperations$,
storedBlockLogs$,
waitForTransaction,
worldContract,
write$: write$.asObservable().pipe(share()),
2 changes: 1 addition & 1 deletion examples/minimal/packages/client-phaser/src/ui/App.tsx
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ export const App = () => {
publicClient: networkLayer.network.publicClient,
walletClient: networkLayer.network.walletClient,
latestBlock$: networkLayer.network.latestBlock$,
blockStorageOperations$: networkLayer.network.blockStorageOperations$,
storedBlockLogs$: networkLayer.network.storedBlockLogs$,
worldAddress: networkLayer.network.worldContract.address,
worldAbi: networkLayer.network.worldContract.abi,
write$: networkLayer.network.write$,
2 changes: 1 addition & 1 deletion examples/minimal/packages/client-react/src/index.tsx
Original file line number Diff line number Diff line change
@@ -24,7 +24,7 @@ setup().then(async (result) => {
publicClient: result.network.publicClient,
walletClient: result.network.walletClient,
latestBlock$: result.network.latestBlock$,
blockStorageOperations$: result.network.blockStorageOperations$,
storedBlockLogs$: result.network.storedBlockLogs$,
worldAddress: result.network.worldContract.address,
worldAbi: result.network.worldContract.abi,
write$: result.network.write$,
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ export async function setupNetwork() {
onWrite: (write) => write$.next(write),
});

const { components, latestBlock$, blockStorageOperations$, waitForTransaction } = await syncToRecs({
const { components, latestBlock$, storedBlockLogs$, waitForTransaction } = await syncToRecs({
world,
config: mudConfig,
address: networkConfig.worldAddress as Hex,
@@ -75,7 +75,7 @@ export async function setupNetwork() {
publicClient,
walletClient: burnerWalletClient,
latestBlock$,
blockStorageOperations$,
storedBlockLogs$,
waitForTransaction,
worldContract,
write$: write$.asObservable().pipe(share()),
Loading