Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(explorer): local indexer inside explorer #3229

Merged
merged 16 commits into from
Sep 25, 2024
Merged

feat(explorer): local indexer inside explorer #3229

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
da3a0f4
initial work for adding indexer to explorer package
karooolis Sep 20, 2024
a5bb435
Merge branch 'main' into kumpis/indexer-inside-explorer
karooolis Sep 24, 2024
772a869
move indexer inside explorer package, update templates
karooolis Sep 24, 2024
2fc0407
rename start store indexer fn
karooolis Sep 25, 2024
aa55ef9
skip store indexer for non-anvil
karooolis Sep 25, 2024
21659a9
fix indexer db path
karooolis Sep 25, 2024
3b42c73
Create silent-hairs-heal.md
karooolis Sep 25, 2024
3ddcd18
remove try/catch
karooolis Sep 25, 2024
7c293cd
Update packages/explorer/src/bin/explorer.ts
karooolis Sep 25, 2024
c83037c
remove indexer db with rm
karooolis Sep 25, 2024
b57c817
STORE_ADDRESS env var, restart indexer on world addr change
karooolis Sep 25, 2024
34c4f75
move to fs/promise variants
karooolis Sep 25, 2024
8f8093d
handle sigterm
karooolis Sep 25, 2024
43e09d7
change to process on exit handler
karooolis Sep 25, 2024
be0af38
Update .changeset/silent-hairs-heal.md
karooolis Sep 25, 2024
6dee218
Merge branch 'main' into kumpis/indexer-inside-explorer
karooolis Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/silent-hairs-heal.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
"@latticexyz/explorer": patch
---

The local store indexer has been integrated into the Explorer package and now runs directly as part of it, without any external setup.
karooolis marked this conversation as resolved.
Show resolved Hide resolved
8 changes: 0 additions & 8 deletions examples/local-explorer/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer --dev
1 change: 1 addition & 0 deletions packages/explorer/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
"@latticexyz/protocol-parser": "workspace:*",
"@latticexyz/schema-type": "workspace:*",
"@latticexyz/store": "workspace:*",
"@latticexyz/store-indexer": "workspace:*",
"@latticexyz/store-sync": "workspace:*",
"@latticexyz/world": "workspace:*",
"@radix-ui/react-checkbox": "^1.1.1",
Expand Down
47 changes: 37 additions & 10 deletions packages/explorer/src/bin/explorer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
#!/usr/bin/env node
import { watchFile } from "fs";
import { readFile } from "fs/promises";
import { readFile, rm } from "fs/promises";
import path from "path";
import process from "process";
import { fileURLToPath } from "url";
import { anvil } from "viem/chains";
import yargs from "yargs";
import { ChildProcess, spawn } from "child_process";
import { validateChainId } from "../common";
Expand Down Expand Up @@ -63,15 +64,18 @@ const argv = yargs(process.argv.slice(2))
.parseSync();

const { port, hostname, chainId, indexerDatabase, worldsFile, dev } = argv;
const indexerDatabasePath = path.join(packageRoot, indexerDatabase);

let worldAddress = argv.worldAddress;
let explorerProcess: ChildProcess;
let indexerProcess: ChildProcess;

async function startExplorer() {
const env = {
...process.env,
CHAIN_ID: chainId.toString(),
WORLD_ADDRESS: worldAddress?.toString(),
INDEXER_DATABASE: path.join(process.cwd(), indexerDatabase),
INDEXER_DATABASE: indexerDatabasePath,
};

if (dev) {
Expand All @@ -97,6 +101,29 @@ async function startExplorer() {
}
}

async function startStoreIndexer() {
if (chainId !== anvil.id) {
console.log("Skipping SQLite indexer for non-anvil chain ID", chainId);
return;
}
karooolis marked this conversation as resolved.
Show resolved Hide resolved

await rm(indexerDatabasePath, { recursive: true, force: true });

console.log("Running SQLite indexer for anvil...");
indexerProcess = spawn("sh", ["node_modules/.bin/sqlite-indexer"], {
cwd: packageRoot,
stdio: "inherit",
env: {
karooolis marked this conversation as resolved.
Show resolved Hide resolved
...process.env,
DEBUG: "mud:*",
RPC_HTTP_URL: "http://127.0.0.1:8545",
FOLLOW_BLOCK_TAG: "latest",
SQLITE_FILENAME: indexerDatabase,
STORE_ADDRESS: worldAddress,
},
});
}

async function readWorldsJson() {
try {
const data = await readFile(worldsFile, "utf8");
Expand All @@ -117,9 +144,10 @@ async function readWorldsJson() {
}

async function restartExplorer() {
if (explorerProcess) {
explorerProcess.kill();
}
indexerProcess?.kill();
explorerProcess?.kill();

await startStoreIndexer();
await startExplorer();
}

Expand All @@ -139,11 +167,9 @@ function watchWorldsJson() {
});
}

process.on("SIGINT", () => {
if (explorerProcess) {
explorerProcess.kill();
}
process.exit();
process.on("exit", () => {
indexerProcess?.kill();
explorerProcess?.kill();
});

async function main() {
Expand All @@ -162,6 +188,7 @@ async function main() {
watchWorldsJson();
}

await startStoreIndexer();
await startExplorer();
}

Expand Down
3 changes: 3 additions & 0 deletions pnpm-lock.yaml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

8 changes: 0 additions & 8 deletions templates/phaser/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer
8 changes: 0 additions & 8 deletions templates/react-ecs/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer
8 changes: 0 additions & 8 deletions templates/react/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer
8 changes: 0 additions & 8 deletions templates/threejs/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer
8 changes: 0 additions & 8 deletions templates/vanilla/mprocs.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,6 @@ procs:
anvil:
cwd: packages/contracts
shell: anvil --base-fee 0 --block-time 2
indexer:
cwd: packages/contracts
shell: shx rm -rf $SQLITE_FILENAME && pnpm sqlite-indexer
env:
DEBUG: mud:*
RPC_HTTP_URL: "http://127.0.0.1:8545"
FOLLOW_BLOCK_TAG: "latest"
SQLITE_FILENAME: "indexer.db"
explorer:
cwd: packages/contracts
shell: pnpm explorer
Loading