Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Version Packages (next) #1921

Merged
merged 1 commit into from
Jan 4, 2024
Merged

Version Packages (next) #1921

merged 1 commit into from
Jan 4, 2024

Conversation

github-actions[bot]
Copy link
Contributor

@github-actions github-actions bot commented Nov 12, 2023 •
edited
Loading

This PR was opened by the Changesets release GitHub action. When you're ready to do a release, you can merge this and the packages will be published to npm automatically. If you're not ready to do a release yet, that's fine, whenever you add more changesets to main, this PR will be updated.

⚠️⚠️⚠️⚠️⚠️⚠️

main is currently in pre mode so this branch has prereleases rather than normal releases. If you want to exit prereleases, run changeset pre exit on main.

⚠️⚠️⚠️⚠️⚠️⚠️

Releases

@latticexyz/[email protected]

Major Changes

  • 85b9461: The postgres indexer is now storing the logIndex of the last update of a record to be able to return the snapshot logs in the order they were emitted onchain.

Minor Changes

  • 1b5eb0d: The findAll method is now considered deprecated in favor of a new getLogs method. This is only implemented in the Postgres indexer for now, with SQLite coming soon. The new getLogs method will be an easier and more robust data source to hydrate the client and other indexers and will allow us to add streaming updates from the indexer in the near future.

    For backwards compatibility, findAll is now implemented on top of getLogs, with record key/value decoding done in memory at request time. This may not scale for large databases, so use wisely.

  • e48fb3b: When the Postgres indexer starts up, it will now attempt to detect if the database is outdated and, if so, cleans up all MUD-related schemas and tables before proceeding.

  • 5df1f31: Added getLogs query support to sqlite indexer

  • 4c1dcd8: - Improved query performance by 10x by moving from drizzle ORM to handcrafted SQL.

    • Moved away from trpc for more granular control over the transport layer.
      Added an /api/logs endpoint using the new query and gzip compression for 40x less data transferred over the wire.
      Deprecated the /trpc/getLogs and /trpc/findAll endpoints.
    • Added a createIndexerClient client for the new /api indexer API exported from @latticexyz/store-sync/indexer-client.
      The createIndexerClient export from @latticexyz/store-sync/trpc-indexer is deprecated.
    - import { createIndexerClient } from "@latticexyz/store-sync/trpc-indexer";
+ import { createIndexerClient } from "@latticexyz/store-sync/indexer-client";

- const indexer = createIndexerClient({ url: "https://indexer.holesky.redstone.xyz/trpc" });
+ const indexer = createIndexerClient({ url: "https://indexer.holesky.redstone.xyz" });

- const snapshot = indexer.getLogs.query(options);
+ const snapshot = indexer.getLogs(options);

  • f61b4bc: The /api/logs indexer endpoint is now returning a 404 snapshot not found error when no snapshot is found for the provided filter instead of an empty 200 response.

Patch Changes

  • 504e25d: Records are now ordered by lastUpdatedBlockNumber at the Postgres SQL query level
  • b00550c: Added a script to run the decoded postgres indexer.
  • 0a3b9b1: Added explicit error logs for unexpected situations.
    Previously all debug logs were going to stderr, which made it hard to find the unexpected errors.
    Now debug logs go to stdout and we can add explicit stderr logs.
  • 85d16e4: Added a Sentry middleware and SENTRY_DNS environment variable to the postgres indexer.
  • c314bad: Replaced Fastify with Koa for store-indexer frontends
  • 392c4b8: Disabled prepared statements for the postgres indexer, which led to issues in combination with pgBouncer.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • 5ab67e3: The error log if no data is found in /api/logs is now stringifying the filter instead of logging [object Object].
  • 735d957: Added a binary for the postgres-decoded indexer.
  • Updated dependencies [5df1f31]
  • Updated dependencies [d8c8f66]
  • Updated dependencies [1b86eac]
  • Updated dependencies [504e25d]
  • Updated dependencies [1077c7f]
  • Updated dependencies [e48fb3b]
  • Updated dependencies [0a3b9b1]
  • Updated dependencies [85b9461]
  • Updated dependencies [a4aff73]
  • Updated dependencies [933b54b]
  • Updated dependencies [712866f]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [6db95ce]
  • Updated dependencies [5d737cf]
  • Updated dependencies [5ac4c97]
  • Updated dependencies [e481717]
  • Updated dependencies [34203e4]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [7eabd06]
  • Updated dependencies [5df1f31]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [7b73f44]

@latticexyz/[email protected]

Major Changes

  • 504e25d: lastUpdatedBlockNumber columns in Postgres storage adapters are no longer nullable

  • e48fb3b: Renamed singleton chain table to config table for clarity.

  • 85b9461: The postgres indexer is now storing the logIndex of the last update of a record to be able to return the snapshot logs in the order they were emitted onchain.

  • a4aff73: Previously, all store-sync strategies were susceptible to a potential memory leak where the stream that fetches logs from the RPC would get ahead of the stream that stores the logs in the provided storage adapter. We saw this most often when syncing to remote Postgres servers, where inserting records was much slower than we retrieving them from the RPC. In these cases, the stream would build up a backlog of items until the machine ran out of memory.

    This is now fixed by waiting for logs to be stored before fetching the next batch of logs from the RPC. To make this strategy work, we no longer return blockLogs# @latticexyz/store-sync (stream of logs fetched from RPC but before they're stored) and instead just return storedBlockLogs# @latticexyz/store-sync (stream of logs fetched from RPC after they're stored).

  • 1b5eb0d: syncToPostgres from @latticexyz/store-sync/postgres now uses a single table to store all records in their bytes form (staticData, encodedLengths, and dynamicData), more closely mirroring onchain state and enabling more scalability and stability for automatic indexing of many worlds.

    The previous behavior, where schemaful SQL tables are created and populated for each MUD table, has been moved to a separate @latticexyz/store-sync/postgres-decoded export bundle. This approach is considered less stable and is intended to be used for analytics purposes rather than hydrating clients. Some previous metadata columns on these tables have been removed in favor of the bytes records table as the source of truth for onchain state.

    This overhaul is considered breaking and we recommend starting a fresh database when syncing with either of these strategies.

  • 7b73f44: Postgres storage adapter now uses snake case for decoded table names and column names. This allows for better SQL ergonomics when querying these tables.

    To avoid naming conflicts for now, schemas are still case-sensitive and need to be queried with double quotes. We may change this in the future with namespace validation.

Minor Changes

  • 5df1f31: Refactored how we fetch snapshots from an indexer, preferring the new getLogs endpoint and falling back to the previous findAll if it isn't available. This refactor also prepares for an easier entry point for adding client caching of snapshots.

    The initialState option for various sync methods (syncToPostgres, syncToRecs, etc.) is now deprecated in favor of initialBlockLogs. For now, we'll automatically convert initialState into initialBlockLogs, but if you want to update your code, you can do:

    import { tablesWithRecordsToLogs } from "@latticexyz/store-sync";

const initialBlockLogs = {
  blockNumber: initialState.blockNumber,
  logs: tablesWithRecordsToLogs(initialState.tables),
};

  • 7eabd06: Added and populated syncProgress key in Zustand store for sync progress, like we do for RECS sync. This will let apps using syncToZustand render a loading state while initial client hydration is in progress.

    const syncProgress = useStore((state) => state.syncProgress);

if (syncProgress.step !== SyncStep.LIVE) {
  return <>Loading ({Math.floor(syncProgress.percentage)}%)</>;
}

  • 4c1dcd8: - Improved query performance by 10x by moving from drizzle ORM to handcrafted SQL.

    • Moved away from trpc for more granular control over the transport layer.
      Added an /api/logs endpoint using the new query and gzip compression for 40x less data transferred over the wire.
      Deprecated the /trpc/getLogs and /trpc/findAll endpoints.
    • Added a createIndexerClient client for the new /api indexer API exported from @latticexyz/store-sync/indexer-client.
      The createIndexerClient export from @latticexyz/store-sync/trpc-indexer is deprecated.
    - import { createIndexerClient } from "@latticexyz/store-sync/trpc-indexer";
+ import { createIndexerClient } from "@latticexyz/store-sync/indexer-client";

- const indexer = createIndexerClient({ url: "https://indexer.holesky.redstone.xyz/trpc" });
+ const indexer = createIndexerClient({ url: "https://indexer.holesky.redstone.xyz" });

- const snapshot = indexer.getLogs.query(options);
+ const snapshot = indexer.getLogs(options);

Patch Changes

  • 0a3b9b1: Added explicit error logs for unexpected situations.
    Previously all debug logs were going to stderr, which made it hard to find the unexpected errors.
    Now debug logs go to stdout and we can add explicit stderr logs.
  • 712866f: createStoreSync now correctly creates table registration logs from indexer records.
  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • 34203e4: Fixed invalid value when decoding records in postgres-decoded storage adapter
  • Updated dependencies [d8c8f66]
  • Updated dependencies [1b86eac]
  • Updated dependencies [1077c7f]
  • Updated dependencies [933b54b]
  • Updated dependencies [f8dab73]
  • Updated dependencies [1a0fa79]
  • Updated dependencies [eb384bb]
  • Updated dependencies [e5a962b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [6db95ce]
  • Updated dependencies [5d737cf]
  • Updated dependencies [5ac4c97]
  • Updated dependencies [e481717]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Minor Changes

  • 59d78c9: Added a mud build command that generates table libraries, system interfaces, and typed ABIs.

Patch Changes

@latticexyz/[email protected]

Minor Changes

  • 1b5eb0d: Added unique and groupBy array helpers to @latticexyz/common/utils.

    import { unique } from "@latticexyz/common/utils";

unique([1, 2, 1, 4, 3, 2]);
// [1, 2, 4, 3]
    import { groupBy } from "@latticexyz/common/utils";

const records = [
  { type: "cat", name: "Bob" },
  { type: "cat", name: "Spot" },
  { type: "dog", name: "Rover" },
];
Object.fromEntries(groupBy(records, (record) => record.type));
// {
//   "cat": [{ type: "cat", name: "Bob" }, { type: "cat", name: "Spot" }],
//   "dog: [{ type: "dog", name: "Rover" }]
// }

  • 4c1dcd8: - Added a Result<Ok, Err> type for more explicit and typesafe error handling (inspired by Rust).

  • 5df1f31: Updated chunk types to use readonly arrays

Patch Changes

  • 933b54b: The benchmark util now logs to stdout instead of stderr.
  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • Updated dependencies [5905420]

[email protected]

Minor Changes

  • f613359: Replaced usage of window global in vanilla JS template with an event listener on the button.

Patch Changes

  • 6963a9e: Templates now correctly include their respective .gitignore files

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [5d737cf]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [5d737cf]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • 9082c17: Updated to use MUD's sendTransaction, which does a better of managing nonces for higher volumes of transactions.
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [5d737cf]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • Updated dependencies [5905420]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [5d737cf]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 9ef3f9a: Fixed an issue where useComponentValue would not detect a change and re-render if the component value was immediately removed.
  • Updated dependencies [d8c8f66]
  • Updated dependencies [1b86eac]
  • Updated dependencies [1077c7f]
  • Updated dependencies [5905420]
  • Updated dependencies [6db95ce]
  • Updated dependencies [5d737cf]
  • Updated dependencies [5ac4c97]
  • Updated dependencies [e481717]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • Updated dependencies [5905420]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.

@latticexyz/[email protected]

Patch Changes

  • d8c8f66: Exclude ERC165 interface ID from custom interface ID's.
  • 1b86eac: Changed the type of the output variable in the slice4 function to bytes4.
  • 1077c7f: Fixed an issue where mud.config.ts source file was not included in the package, causing TS errors downstream.
  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • 6db95ce: Fixed StoreCore to pass previousEncodedLengths into onBeforeSpliceDynamicData.
  • 5d737cf: Updated the debug util to pipe to stdout and added an additional util to explicitly pipe to stderr when needed.
  • 5ac4c97: Fixed M-04 Memory Corruption on Load From Storage
    It only affected external use of Storage.load with a memoryPointer argument
  • e481717: Removed unused imports from various files in the store and world packages.
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [5d737cf]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.

@latticexyz/[email protected]

Patch Changes

  • d8c8f66: Exclude ERC165 interface ID from custom interface ID's.
  • 1077c7f: Fixed an issue where mud.config.ts source file was not included in the package, causing TS errors downstream.
  • f8dab73: Added explicit internal visibility to the coreSystem variable in CoreModule.
  • 1a0fa79: Fixed requireInterface to correctly specify ERC165.
  • eb384bb: Added isInstalled and requireNotInstalled helpers to Module base contract.
  • e5a962b: World now correctly registers the FunctionSignatures table.
  • 5905420: TS packages now generate their respective .d.ts type definition files for better compatibility when using MUD with moduleResolution set to bundler or node16 and fixes issues around missing type declarations for dependent packages.
  • e481717: Removed unused imports from various files in the store and world packages.
  • Updated dependencies [d8c8f66]
  • Updated dependencies [1b86eac]
  • Updated dependencies [1077c7f]
  • Updated dependencies [933b54b]
  • Updated dependencies [5905420]
  • Updated dependencies [1b5eb0d]
  • Updated dependencies [6db95ce]
  • Updated dependencies [5d737cf]
  • Updated dependencies [5ac4c97]
  • Updated dependencies [e481717]
  • Updated dependencies [4c1dcd8]
  • Updated dependencies [5df1f31]

@latticexyz/[email protected]

Patch Changes

[email protected]

[email protected]

@latticexyz/[email protected]

@latticexyz/[email protected]

@latticexyz/[email protected]

@latticexyz/[email protected]

@latticexyz/[email protected]

@latticexyz/[email protected]

@github-actions github-actions bot requested review from alvrs and holic as code owners November 12, 2023 08:30
@github-actions github-actions bot force-pushed the changeset-release/main branch from f4a633f to cb9b313 Compare November 13, 2023 07:02
@github-actions github-actions bot force-pushed the changeset-release/main branch from cb9b313 to 45247db Compare November 13, 2023 07:20
@github-actions github-actions bot force-pushed the changeset-release/main branch from 45247db to 0e32c32 Compare November 13, 2023 07:29
@github-actions github-actions bot force-pushed the changeset-release/main branch from 0e32c32 to 55f374a Compare November 13, 2023 07:37
@github-actions github-actions bot force-pushed the changeset-release/main branch from 55f374a to fa0b8ef Compare November 13, 2023 07:43
@github-actions github-actions bot force-pushed the changeset-release/main branch from fa0b8ef to 4f563c8 Compare November 13, 2023 07:50
@github-actions github-actions bot force-pushed the changeset-release/main branch from 4f563c8 to 3df6443 Compare November 13, 2023 07:55
@github-actions github-actions bot force-pushed the changeset-release/main branch from 3df6443 to 1a5d1b3 Compare November 13, 2023 08:08
@github-actions github-actions bot force-pushed the changeset-release/main branch from 1a5d1b3 to 07921af Compare November 13, 2023 08:14
@github-actions github-actions bot force-pushed the changeset-release/main branch from 2a370d0 to 8fb6b11 Compare January 2, 2024 10:24
@github-actions github-actions bot force-pushed the changeset-release/main branch from 8fb6b11 to bf10d53 Compare January 2, 2024 10:52
@github-actions github-actions bot force-pushed the changeset-release/main branch from bf10d53 to 571dd25 Compare January 2, 2024 12:30
@github-actions github-actions bot force-pushed the changeset-release/main branch from 571dd25 to 790ccda Compare January 2, 2024 14:39
@github-actions github-actions bot force-pushed the changeset-release/main branch from 790ccda to 350e177 Compare January 2, 2024 15:48
@github-actions github-actions bot force-pushed the changeset-release/main branch from 350e177 to 2b74f34 Compare January 2, 2024 15:56
@github-actions github-actions bot force-pushed the changeset-release/main branch from 2b74f34 to c9b36e0 Compare January 3, 2024 11:26
@github-actions github-actions bot force-pushed the changeset-release/main branch from c9b36e0 to 02d33f5 Compare January 3, 2024 14:38
@github-actions github-actions bot force-pushed the changeset-release/main branch from 02d33f5 to 92cd707 Compare January 3, 2024 14:51
@github-actions github-actions bot force-pushed the changeset-release/main branch from 92cd707 to c3a588e Compare January 3, 2024 14:58
@github-actions github-actions bot force-pushed the changeset-release/main branch from c3a588e to 95b8348 Compare January 3, 2024 15:05
@github-actions github-actions bot force-pushed the changeset-release/main branch from 95b8348 to 0241ec0 Compare January 3, 2024 17:50
@github-actions github-actions bot force-pushed the changeset-release/main branch from 0241ec0 to a9f5e70 Compare January 3, 2024 17:58
@alvrs alvrs merged commit 6780dd9 into main Jan 4, 2024
1 check passed
@alvrs alvrs deleted the changeset-release/main branch January 4, 2024 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alvrs alvrs alvrs approved these changes

@holic holic Awaiting requested review from holic holic is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@alvrs