Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store-sync,store-indexer): add postgres support #1338

Merged
merged 38 commits into from
Sep 1, 2023

Conversation

holic
Copy link
Member

@holic holic commented Aug 21, 2023

fixes #1193

Mostly adapted from (i.e. copy pasted) sqlite -> postgres. The main changes are in handling schemas/namespaces (which sqlite doesn't have). I did a bit of refactoring too.

I left it as mostly duplicate code for now until we can figure out good abstractions. Drizzle makes it tricky because there aren't really shared types between types like SqliteTable and PgTable. And executing queries is different for each flavor/dialect (sync vs async, run vs execute).

@changeset-bot
Copy link

changeset-bot bot commented Aug 21, 2023

⚠️ No Changeset found

Latest commit: 03e5e69

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@holic holic changed the title WIP postgres support feat(store-sync,store-indexer): add postgres support Aug 23, 2023
@holic holic marked this pull request as ready for review August 23, 2023 14:23
@holic holic requested a review from alvrs as a code owner August 23, 2023 14:23
"start": "tsx bin/postgres-indexer",
"start:local": "DATABASE_URL=http://127.0.0.1/postgres CHAIN_ID=31337 pnpm start",
"start:testnet": "DATABASE_URL=http://127.0.0.1/postgres CHAIN_ID=4242 START_BLOCK=19037160 pnpm start",
"start:testnet2": "DATABASE_URL=http://127.0.0.1/postgres CHAIN_ID=4243 pnpm start",
Copy link
Member Author

@holic holic Aug 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any opinions on whether we should keep this defaulting to sqlite and have separate scripts for postgres? sqlite feels a bit better for a quick-to-spin-up local thing (less overhead than e.g. ensuring postgres is running)

@holic holic force-pushed the holic/postgres-storage-adapter branch from 7de9c6b to 83abbdf Compare August 23, 2023 15:36
alvrs
alvrs previously approved these changes Aug 30, 2023
Copy link
Member

@alvrs alvrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking great! just a couple non-blocking nits/questions

@@ -54,14 +65,21 @@ describe("Sync from indexer", async () => {
expect(asyncErrorHandler.getErrors()[0]).toContain("error fetching initial state from indexer");
});

describe("indexer online", () => {
describe.each([["sqlite"], ["postgres"]] as const)("%s indexer", (indexerType) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very cool 👍

@@ -6,6 +6,9 @@
"scripts": {
"test": "vitest --run"
},
"dependencies": {
"zod": "^3.22.2"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i think this could be a dev dependency?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh sure, I guess the whole package is ~dev so I wasn't paying too close attention to it

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it's not important, just OCD inducing that one dependency is not marked as dev

},
});

await server.listen({ port: env.PORT });
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we make the host configurable here or do you want to update that for both sqlite and postgres indexer in a followup PR? (like e70bd42)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh yep, can add that here

operations: StorageOperation<TConfig>[];
}) => Promise<void>;
};
export type BlockLogsToStorageOptions<TConfig extends StoreConfig = StoreConfig> = StorageAdapter<TConfig>;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need BlockLogsToStorageOptions as separate type? Could we just use StorageAdapter everywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is just leftover from a refactor! Will clean up.

I am starting to prefer these explicit params+return types even if it's just a simple alias like this. But I don't feel that strongly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removing this revealed a lot more clean up work, so good call! feels much better now without it

Comment on lines +11 to +12
// TODO: change schema version to varchar/text?
schemaVersion: integer("schema_version").notNull().primaryKey(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this already in preparation for a protocol type?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's a reflection of the schemaVersion.ts file (we have this in the sqlite indexer already) to help detect when the indexer meta tables change (the shape of the indexer DB as opposed to the shape of the MUD tables) so we can recreate them

I opened an issue for a better approach: #1362

alvrs
alvrs previously approved these changes Sep 1, 2023
@holic holic merged commit b77b1a4 into main Sep 1, 2023
@holic holic deleted the holic/postgres-storage-adapter branch September 1, 2023 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for Postgres in sync stack
2 participants