Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(store-sync): snake case postgres names in decoded tables #1989

Merged
merged 5 commits into from
Dec 1, 2023

Conversation

holic
Copy link
Member

@holic holic commented Dec 1, 2023

fixes #1987

Copy link

changeset-bot bot commented Dec 1, 2023

🦋 Changeset detected

Latest commit: ab29b8b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 30 packages
Name Type
@latticexyz/store-sync Major
@latticexyz/dev-tools Major
@latticexyz/store-indexer Major
@latticexyz/abi-ts Major
@latticexyz/block-logs-stream Major
@latticexyz/cli Major
@latticexyz/common Major
@latticexyz/config Major
create-mud Major
@latticexyz/ecs-browser Major
@latticexyz/faucet Major
@latticexyz/gas-report Major
@latticexyz/network Major
@latticexyz/noise Major
@latticexyz/phaserx Major
@latticexyz/protocol-parser Major
@latticexyz/react Major
@latticexyz/recs Major
@latticexyz/schema-type Major
@latticexyz/services Major
@latticexyz/solecs Major
solhint-config-mud Major
solhint-plugin-mud Major
@latticexyz/std-client Major
@latticexyz/std-contracts Major
@latticexyz/store-cache Major
@latticexyz/store Major
@latticexyz/utils Major
@latticexyz/world-modules Major
@latticexyz/world Major

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@holic holic changed the title Holic/indexer snake case feat(store-sync): snake case postgres names Dec 1, 2023
@holic holic changed the base branch from main to holic/clean-database December 1, 2023 11:23
// (no need to double-quote), but this is destructive and may lead to
// overlapping namespaces (both `MyNamespace` and `my_namespace` snake case
// to the same value).
// TODO: consider not snake casing these
Copy link
Member Author

@holic holic Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alvrs Curious if you have thoughts on this!

This snake case approach was inspired by Sky Strife analytics queries, where the SQL is quite ugly due to all the double quotes for case sensitivity:
https://github.com/latticexyz/mud/compare/historical-indexer?expand=1#diff-ec8025f1e682cd379feaad115d8f46b1220fc4a37e1859dc9a360e8b54fd9d14

But for schema name and table name, it's a bit of a hairy trade-off in that this could cause naming overlaps and errors.

(We already have the potential for issues here because there's no guarantee a table ID is actually a nice alphanumeric string and could be a bunch of binary and therefore wouldn't translate well to a schema or table name anyway)

Copy link
Member Author

@holic holic Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted to only snake case column names, the best we can do with SQL for case sensitive schemas/tables is something like

SET search_path TO "0x7203e7ADfDF38519e1ff4f8Da7DCdC969371f377__";

SELECT
    __last_updated_block_number AS created_at_block,
    created_by,
    encode(level_id, 'escape') AS map,
    key AS match_entity
FROM
    "MatchConfig" match_config;
  • set search_path to the schema(s) we want to query from with their "shorthand" form (i.e. just table name)
  • manually alias case-sensitive MatchConfig table name to match_config, in case we want reference it in joins without double quotes
  • assumes column names are snake cased (safe IMO)

For ORMs and other programmatic SQL tools, I'm not sure how easy it is to use something like search_path, but maybe these would also abstract over the quoting of things like schemas to ensure case sensitivity.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in general we might have to add an onchain verification of only alphanumeric characters in namespaces and name (also to avoid conflicts for namespace A_B name C with namespace A name B_C, and for the reasons you mentioned). If we have that guarantee from the protocol side, we can also do the automatic snake case conversion here. Until then I think using the search_path approach might be the better temporary solution (so not doing the snake casing of table id yet to avoid conflict issues for now)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added an issue for validating namespaces: #1991

Worth thinking about whether this sync/indexer should be World-aware or only Store-aware. I think the bytes one is only Store-aware and it doesn't care about namespaces, etc. But the decoded one does, and therefore is more of a "world sync" than a "store sync".

@holic holic marked this pull request as ready for review December 1, 2023 11:27
@holic holic requested a review from alvrs as a code owner December 1, 2023 11:27
@holic holic changed the title feat(store-sync): snake case postgres names feat(store-sync): snake case postgres names Dec 1, 2023
@holic holic changed the title feat(store-sync): snake case postgres names feat(store-sync): snake case postgres names in decoded tables Dec 1, 2023
Base automatically changed from holic/clean-database to main December 1, 2023 13:51
@holic holic merged commit 7b73f44 into main Dec 1, 2023
10 checks passed
@holic holic deleted the holic/indexer-snake-case branch December 1, 2023 14:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

postgres indexer: consider snake-case for table/column names
2 participants