From 8d3f1f4c857cf38f99d1ea4cee7655136598bb45 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Mon, 16 Oct 2023 13:57:17 -0500 Subject: [PATCH 1/8] WIP --- next-docs/pages/services/indexer.mdx | 518 +++++++++++++++++++++++++++ 1 file changed, 518 insertions(+) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index e2acd194b8..3c2ee3ff69 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -1 +1,519 @@ +import WarningBox from "../../components/WarningBox"; + # Indexer + +The MUD Indexer is an off-chain indexer for on-chain applications built with MUD. + +## Why an off-chain indexer? + +Reads with on-chain apps can be tricky. +What does it mean to be able to query the Ethereum network? +Technically, given a node with a fully synced state, we can explore just about everything using the EVM, but the “exploring” would be looking at raw storage slots for accounts corresponding to smart contracts. +A way around this exists by providing view functions on smart contracts: these effectively are just wrappers around raw storage and expose a more friendly API. +Instead of having to figure out where the balances for an account are stored in the storage tree, we now can call a function that does the lookup via Solidity via an RPC endpoint. + +The issue with view functions is that complexity in having to generate and call these to get the “full picture” of the state from the chain explodes pretty quickly. +For a web app client wanting to present the user with an up-to-date view of the on-chain state, the task of calling the different view functions becomes unnecessarily hard very fast. +This can also quickly accelerate the need to run a set of dedicated nodes just to be able to service the needed number of RPC requests. + +## Installation + +### Running the indexer locally (on the host) + +These are the steps to install and start the MUD indexer. +They are written under the assumption you are using `anvil` for your test chain, which is what the getting started package's `pnpm dev` does. + +1. Build the repository [as explained here](contribute#local-development-setup). + +1. Ensure you have [`jq`](https://jqlang.github.io/jq/) installed. + +1. Run a test world. + The easiest way to do this is to follow [these directions](tutorials/minimal#getting-the-initial-version) in a separate command line window. + + **Note**: While MUD v2 is in alpha, there might be breaking changes between pre-release versions. + To ensure your World is compatible with your indexer, make sure both are running the same MUD version. + To create a project with the MUD version on the `main` branch, use `mud@main` instead of ~`mud@next`~. + + ```sh copy + pnpm create mud@main project-name + ``` + +1. Start the indexer. + + ```sh copy + cd packages/store-indexer + pnpm start:sqlite:local + ``` + + Note: If you have already ran the MUD indexer, and then restarted the blockchain, make sure to delete the old database: + + ```sh copy + rm anvil.db + ``` + +1. Run this command (in a separate command line window) as a sanity check to verify the indexer is working correctly: + + ```sh copy + curl 'http://localhost:3001/trpc/findAll?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' | jq + ``` + + The result should be nicely formatted (and long) JSON output. + +### Running the indexer on Docker + +The indexer Docker image is available [on github](https://github.com/latticexyz/mud/pkgs/container/store-indexer). + +These environment variables need to be provided to the docker indexer to work: + +| Type | Variable | Meaning | Sample value (using `anvil` running on the host) | +| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | +| Needed | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 | +| Optional | RPC_WS_URL | The URL to access the blockchain using WebSocket | +| Optional | START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | 1 | +| Optional | DEBUG=mud:\* | Turn on debugging | | +| Optional, only for SQLite | SQLITE_FILENAME | Name of database | `anvil.db` | +| Optional, only for PostgreSQL | DATABASE_URL | URL for the database, of the form `postgres:///` | + +There are several ways to provide those environment variables to `docker run`: + +- On the command line you can specify `-e =`. + You specify this after the `docker run`, but before the name of the image. +- You can also write all the environment variables in a file and specify it using `--env-file`. + You specify this after the `docker run`, but before the name of the image. +- Both [Docker Compose](https://docs.docker.com/compose/) and [Kubernetes](https://kubernetes.io/) have their own mechanisms for starting docker containers with environment variables. + +#### SQLite + +The command to start the indexer in SQLite mode is `pnpm start:sqlite`. +To index an `anvil` instance running to the host, use: + +```sh copy +docker run \ + --platform linux/amd64 \ + -e RPC_HTTP_URL=http://host.docker.internal:8545 \ + -p 3001:3001 \ + ghcr.io/latticexyz/store-indexer:latest \ + pnpm start:sqlite +``` + +However, this creates a docker container with a state, the SQLite database file. +If we start a new container with the same image and parameters, it is going to have to go back to the start of the blockchain, which depending on how long the blockchain has been in use may be a problem. +We can solve this with [volumes](https://docs.docker.com/storage/volumes/): + +1. Create a docker volume for the SQLite database file. + + ```sh copy + docker volume create sqlite-db-file + ``` + +1. Run the indexer container using the volume. + + ```sh copy + docker run \ + --platform linux/amd64 \ + -e RPC_HTTP_URL=http://host.docker.internal:8545 \ + -e SQLITE_FILENAME=/dbase/anvil.db \ + -v sqlite-db-file:/dbase \ + -p 3001:3001 \ + ghcr.io/latticexyz/store-indexer:latest \ + pnpm start:sqlite + ``` + +1. You can stop the docker container and restart it, or start a separate container using the same database. + +1. When you are done, you have to delete the docker containers that used it before you can delete the volume. + You can use these commands: + + ```sh copy + docker rm `docker ps -a --filter volume=sqlite-db-file -q` + docker volume rm sqlite-db-file + ``` + + **Note:** You should do this every time you restart the blockchain. + Otherwise your index will include data from multiple blockchains, and make no sense. + +#### PostgreSQL + +The command to start the indexer in PostgreSQL mode is `pnpm start:postgres`. + +1. The docker instance identifies to PostgreSQL as `root`. + To give this user permissions on the database, enter `psql` and run this command. + + ```sql copy + CREATE ROLE root SUPERUSER LOGIN; + ``` + + **Note:** This is assuming a database that is isolated from the Internet and only used by trusted entities. + In a production system you will use at least a password as authentication, and limit the user's authority. + +1. Start the docker container. + For example, to index an `anvil` instance running to the host to the database `postgres` on the host, use. + + ```sh copy + docker run \ + --platform linux/amd64 \ + -e RPC_HTTP_URL=http://host.docker.internal:8545 \ + -e DATABASE_URL=postgres://host.docker.internal/postgres \ + -p 3001:3001 \ + ghcr.io/latticexyz/store-indexer:latest \ + pnpm start:postgres + ``` + +1. The PostgreSQL database is persistent. + If you restart the blockchain you have to delete the content of this database, otherwise the indexer will include old information. + To delete the database content, enter `psql` and run this command. + + ```sql copy + DROP OWNED BY root; + ``` + +## Data access and interpretation + +### Using the indexer with TypeScript + +You can see [an example of an indexer client in the MUD repo](https://github.com/latticexyz/mud/tree/main/examples/indexer-client). +To execute it: + +1. Download and build the example. + + ```sh copy + git clone https://@github.com/latticexyz/mud.git + cd mud/examples/indexer-client + pnpm install + ``` + +1. Run the example. + + ```sh copy + pnpm read-indexer + ``` + +If your indexer is indexing [the minimal template](/tutorials/minimal), the expected output should be similar to: + +``` +Block number: 2042 +Tables: schema,Hooks,StoreMetadata,NamespaceOwner,InstalledModules,ResourceAccess,Systems,FunctionSelector,SystemHooks,SystemRegistry,ResourceType,Counter +Information about Counter +{ + id: '0x5FbDB2315678afecb367f032d93F642f64180aa3____Counter', + address: '0x5FbDB2315678afecb367f032d93F642f64180aa3', + tableId: '0x00000000000000000000000000000000436f756e746572000000000000000000', + namespace: '', + name: 'Counter', + keySchema: {}, + valueSchema: { value: 'uint32' }, + lastUpdatedBlockNumber: 2042n, + records: [ { key: {}, value: [Object] } ] +} +The actual value: 2 +``` + +#### Detailed explanation + +```ts +import { createIndexerClient } from "@latticexyz/store-sync/trpc-indexer"; +const indexer = createIndexerClient({ + url: "http://127.0.0.1:3001/trpc", +}); +``` + +Create an indexer client. The URL, `http://127.0.0.1/3001/trpc`, is the one for the indexer you run on the local computer. +If the indexer is elsewhere, modify the URL as appropriate. + +```ts +const result = await indexer.findAll.query({ + chainId: 31337, + address: "0x5FbDB2315678afecb367f032d93F642f64180aa3", +}); +``` + +Connect to the indexer. +The `chainId` value is for a sanity check, to ensure we are connecting to the correct indexer. +The `address` parameter is the address of the `World` contract whose information we are reading. +The same indexer can read from multiple `World` objects, as long as they are on the same blockchain. + +```ts +console.log(`Block number: ${result.blockNumber}`); +``` + +The `result` has two fields: + +- `blockNumber`, the data is correct as of that block (it could change later). +- `tables`, the tables of the `World` we are reading. + +```ts +console.log(`Tables: ${result.tables.map((t) => t.name)}`); +``` + +Use [`map`](https://www.tutorialspoint.com/typescript/typescript_array_map.htm) to get the names of the tables. + +```ts +const counterTable = result.tables.filter((t) => t.name == "Counter")[0]; +``` + +Use [`filter`](https://www.tutorialspoint.com/typescript/typescript_array_filter.htm) to get the `Counter` table, assuming there is at least one. + +```ts +console.log("Information about Counter"); +console.log(counterTable); +``` + +Log the information we have about the table. +It contains these fields: + +| Field | Meaning | +| ---------------------- | ------------------------------------------------------------------------------------- | +| id | Iinternal ID for uniqueness in the context of SQL | +| address | `World` address | +| tableId | `bytes32` hex encoded table ID (a concat of `bytes16(namespace)` and `bytes16(name)`) | +| namespace | Table's namespace | +| name | Table's name | +| keySchema | Schema of the table key | +| valueSchema | Schema of the table value | +| lastUpdatedBlockNumber | Block number for which the data is correct | +| records | Actual table data | + +```ts +console.log(`The actual value: ${counterTable.records[0].value.value}`); +``` + +The actual value in `Counter`, which is in the first (and only) record. +Every record has two fields: + +- `key` +- `value` + +Both `key` and `value` can have multiple fields inside them, one for each field in the appropriate schema. +In the case of `Counter` there is nothing in the `key`, but `value` has a single field, also called `value`. +So the counter value is `value.value` of the record. + +### Using HTTP + + +

+ The parameters, especially the URL, may change as we adjust our tRPC configuration. + + If you can use the TypeScript methods, those are more stable. +

+ + +} +/> + +The indexer is built on top of [tRPC](https://trpc.io/), which is primarily designed to be used between [TypeScript](https://www.typescriptlang.org/) programs. +As a result, the data format is more complicated than REST APIs. + +For queries, the indexer expects to get two parameters in a [query string](https://en.wikipedia.org/wiki/Query_string): + +- `batch`, the batch identifier. + This parameter can contain any string, as long as it is present. + +- `input`, the actual query. + This query is a [URLEncoded](https://en.wikipedia.org/wiki/Query_string#URL_encoding) [JSON](https://en.wikipedia.org/wiki/JSON) object. + To play with possible object values, you can use [this online calculator](https://www.urlencoder.org/). + + For example, the input object used in the sanity check above is: + + ```json + { + "0": { + "json": { + "chainId": 31337, + "address": "0x5FbDB2315678afecb367f032d93F642f64180aa3" + } + } + } + ``` + + An HTTP request can include multiple queries, so the query object includes a numbered key (or numbered keys) with the actual queries. + Also, in the future other formats than JSON might be supported, so each query specifies what format it uses. + +The one type of query currently supported is `findAll`. It gives you all the information stored on the specific world in the indexer. +The query is available at path `/findAll` and have two parameters: + +- `chainId`: The chain ID of the blockchain to query. +- `address`: The address of the `World`. + Based on the paramters in the development environment, `pnpm dev` uses the address in the sanity check. + +To understand the result, it is easiest to look at it inside Node. + +1. Read the result and enter `node`: + + ```sh copy + curl 'http://localhost:3001/findAll?batch=wd&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' > res.json + node + ``` + +1. Read the result into a JavaScript object. + + ```javascript copy + res = JSON.parse(fs.readFileSync("res.json")); + ``` + +1. `res` is actually a list of results, which only has one item because we only had one key in the query. + This item has one key, `record`, whose value also has only one key, `data`. + Run these commands to verify the statements above and then cut `res` to the actual result. + + ```javascript copy + res.length; + Object.keys(res[0]); + Object.keys(res[0].result); + res = res[0].result.data; + ``` + +1. Inside the result there are two fields: + + - `meta`, the [metadata](https://en.wikipedia.org/wiki/Metadata). + In the case of JSON the field names are part of the data, so this primarily contains data types. + - `json`, the data read from the `World`. + This field itself is divided into two values: + - `blockNumber`, the block number in which these query results apply. + - `tables`, a list of the tables in the `World` and their data. + + You can use [the `filter` function](https://www.w3schools.com/jsref/jsref_filter.asp) to get a specific table. + In this case, we are getting the `Counter` table. + + ```javascript copy + counterTable = res.json.tables.filter((t) => t.name === "Counter")[0]; + ``` + +1. Finally, to get the actual record with the counter value, use: + + ```javascript copy + counterTable.records[0].value.value; + ``` + +## Debugging + +[See here](/reference#mud-server-components). + +## Indexing other blockchains + +To index a different blockchain, you specify these environment variables: + +| Variable | Value | +| ----------------------- | -------------------------------------------------------------------------------------------------------- | +| RPC_HTTP_URL | The URL to access the blockchain using HTTP | +| RPC_WS_URL | The URL to access the blockchain using WebSocket | +| START_BLOCK1 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | +| SQLITE_FILENAME | `.db` | + +(1) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed. + +After you do that, start the indexer with this command: + +```sh copy +pnpm start:sqlite +``` + +Note that in your queries you need to specify the correct chainId for the chain you are indexing. + +## Using the indexer with PostgreSQL + +To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these steps: + +1. Download and install [PostgreSQL](https://www.postgresql.org/download/). + +1. To use the default setting (local PostgreSQL, using the `postgres` database, indexing the local `anvil` chain), run: + + ```sh copy + pnpm start:postgres:local + ``` + + Otherwise, set these environment variables: + + | Variable | Value | + | ------------------------ | -------------------------------------------------------------------------------------------------------- | + | CHAIN_ID | ChainID for the chain you index | + | RPC_HTTP_URL1 | The URL to access the blockchain using HTTP | + | RPC_WS_URL1 | The URL to access the blockchain using WebSocket | + | START_BLOCK2 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | + | DATABASE_URL | URL for the database, of the form `postgres:///` | + + (1) Not necessary in the case of a Lattice testnet. + + (2) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed for performance reasons. + + And then run: + + ```sh copy + pnpm start:postgres + ``` + +### Reading information from PostgreSQL + +You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) to view the indexer information. + +Note: This example uses [the template](http://localhost:3000/tutorials/minimal). +If you are indexing a different `World`, change the world address and use the name of a table you have rather than `Counter`. + +1. Get the list of schemas to verify `__mud_internal` is there. + + ```sql copy + \dn + ``` + +1. Get the list of relations in the schema. + + ```sql copy + \dt __mud_internal.* + ``` + + You should get two tables, `chain` and `tables`. + +1. Get the list of chains being indexed. + + ```sql copy + SELECT * FROM __mud_internal.chain; + ``` + +1. The `tables` table is a lot bigger, so first get the list of columns within the table. + + ```sql copy + \d+ __mud_internal.tables + ``` + +1. Get the list of MUD table names. + + ```sql copy + SELECT name FROM __mud_internal.tables; + ``` + +1. Get the key and value schemas for `Counter`. + + ```sql copy + SELECT key_schema, value_schema FROM __mud_internal.tables WHERE name='Counter'; + ``` + +1. The schema for a world is `
__`. + Get the list of tables for a specific world. + + ```sql copy + \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* + ``` + +1. Specify the world schema is the default. + + ```sql copy + SET search_path TO "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; + ``` + +1. Get the value of the counter. + + ```sql copy + SELECT * FROM "Counter"; + ``` + +### Deleting the PostgreSQL data + +PostgreSQL is a persistent data storage. +This means that if you restart the blockchain without manually deleting the data, your indexer will use old data and provide incorrect results. + +To avoid this, enter `psql` and run this command: + +```sql copy +DROP OWNED BY ; +``` From 29df1ec1b19fbf3aa861d1ffbfbba3d8a76e8ad8 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Tue, 17 Oct 2023 10:17:23 -0500 Subject: [PATCH 2/8] WIP --- next-docs/pages/services/indexer.mdx | 423 ++------------------------- 1 file changed, 28 insertions(+), 395 deletions(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index 3c2ee3ff69..5e7c6747db 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -18,63 +18,39 @@ This can also quickly accelerate the need to run a set of dedicated nodes just t ## Installation -### Running the indexer locally (on the host) +These environment variables need to be provided to the indexer to work: -These are the steps to install and start the MUD indexer. -They are written under the assumption you are using `anvil` for your test chain, which is what the getting started package's `pnpm dev` does. - -1. Build the repository [as explained here](contribute#local-development-setup). +| Type | Variable | Meaning | Sample value (using `anvil` running on the host) | +| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | +| Needed | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 | +| Optional | RPC_WS_URL | The URL to access the blockchain using WebSocket | +| Optional | START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | 1 | +| Optional | DEBUG=mud:\* | Turn on debugging | | +| Optional, only for SQLite | SQLITE_FILENAME | Name of database | `anvil.db` | +| Optional, only for PostgreSQL | DATABASE_URL | URL for the database, of the form `postgres:///` | -1. Ensure you have [`jq`](https://jqlang.github.io/jq/) installed. +### Running directly -1. Run a test world. - The easiest way to do this is to follow [these directions](tutorials/minimal#getting-the-initial-version) in a separate command line window. +To run the indexer directly on your computer: - **Note**: While MUD v2 is in alpha, there might be breaking changes between pre-release versions. - To ensure your World is compatible with your indexer, make sure both are running the same MUD version. - To create a project with the MUD version on the `main` branch, use `mud@main` instead of ~`mud@next`~. +1. Create a `.env` file with the environment variables. + For example, if you are running the template with `pnpm dev` locally, you might use this file: - ```sh copy - pnpm create mud@main project-name + ```bash filename=".env" + RPC_HTTP_URL=http://localhost:8545 ``` -1. Start the indexer. +1. Install the indexer - ```sh copy - cd packages/store-indexer - pnpm start:sqlite:local - ``` + ```bash - Note: If you have already ran the MUD indexer, and then restarted the blockchain, make sure to delete the old database: - - ```sh copy - rm anvil.db ``` -1. Run this command (in a separate command line window) as a sanity check to verify the indexer is working correctly: - - ```sh copy - curl 'http://localhost:3001/trpc/findAll?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' | jq - ``` - - The result should be nicely formatted (and long) JSON output. - -### Running the indexer on Docker +### Docker The indexer Docker image is available [on github](https://github.com/latticexyz/mud/pkgs/container/store-indexer). -These environment variables need to be provided to the docker indexer to work: - -| Type | Variable | Meaning | Sample value (using `anvil` running on the host) | -| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -| Needed | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 | -| Optional | RPC_WS_URL | The URL to access the blockchain using WebSocket | -| Optional | START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | 1 | -| Optional | DEBUG=mud:\* | Turn on debugging | | -| Optional, only for SQLite | SQLITE_FILENAME | Name of database | `anvil.db` | -| Optional, only for PostgreSQL | DATABASE_URL | URL for the database, of the form `postgres:///` | - -There are several ways to provide those environment variables to `docker run`: +There are several ways to provide the environment variables to `docker run`: - On the command line you can specify `-e =`. You specify this after the `docker run`, but before the name of the image. @@ -82,6 +58,8 @@ There are several ways to provide those environment variables to `docker run`: You specify this after the `docker run`, but before the name of the image. - Both [Docker Compose](https://docs.docker.com/compose/) and [Kubernetes](https://kubernetes.io/) have their own mechanisms for starting docker containers with environment variables. +The easiest way to test the indexer is to [run the template as a world]() in a separate command-line window. + #### SQLite The command to start the indexer in SQLite mode is `pnpm start:sqlite`. @@ -119,6 +97,12 @@ We can solve this with [volumes](https://docs.docker.com/storage/volumes/): pnpm start:sqlite ``` +1. Run this command to test next indexer. + + ```sh copy + curl 'http://localhost:3001/trpc/findAll?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' | jq + ``` + 1. You can stop the docker container and restart it, or start a separate container using the same database. 1. When you are done, you have to delete the docker containers that used it before you can delete the volume. @@ -136,7 +120,7 @@ We can solve this with [volumes](https://docs.docker.com/storage/volumes/): The command to start the indexer in PostgreSQL mode is `pnpm start:postgres`. -1. The docker instance identifies to PostgreSQL as `root`. +1. The docker instance identifies itself to PostgreSQL as `root`. To give this user permissions on the database, enter `psql` and run this command. ```sql copy @@ -166,354 +150,3 @@ The command to start the indexer in PostgreSQL mode is `pnpm start:postgres`. ```sql copy DROP OWNED BY root; ``` - -## Data access and interpretation - -### Using the indexer with TypeScript - -You can see [an example of an indexer client in the MUD repo](https://github.com/latticexyz/mud/tree/main/examples/indexer-client). -To execute it: - -1. Download and build the example. - - ```sh copy - git clone https://@github.com/latticexyz/mud.git - cd mud/examples/indexer-client - pnpm install - ``` - -1. Run the example. - - ```sh copy - pnpm read-indexer - ``` - -If your indexer is indexing [the minimal template](/tutorials/minimal), the expected output should be similar to: - -``` -Block number: 2042 -Tables: schema,Hooks,StoreMetadata,NamespaceOwner,InstalledModules,ResourceAccess,Systems,FunctionSelector,SystemHooks,SystemRegistry,ResourceType,Counter -Information about Counter -{ - id: '0x5FbDB2315678afecb367f032d93F642f64180aa3____Counter', - address: '0x5FbDB2315678afecb367f032d93F642f64180aa3', - tableId: '0x00000000000000000000000000000000436f756e746572000000000000000000', - namespace: '', - name: 'Counter', - keySchema: {}, - valueSchema: { value: 'uint32' }, - lastUpdatedBlockNumber: 2042n, - records: [ { key: {}, value: [Object] } ] -} -The actual value: 2 -``` - -#### Detailed explanation - -```ts -import { createIndexerClient } from "@latticexyz/store-sync/trpc-indexer"; -const indexer = createIndexerClient({ - url: "http://127.0.0.1:3001/trpc", -}); -``` - -Create an indexer client. The URL, `http://127.0.0.1/3001/trpc`, is the one for the indexer you run on the local computer. -If the indexer is elsewhere, modify the URL as appropriate. - -```ts -const result = await indexer.findAll.query({ - chainId: 31337, - address: "0x5FbDB2315678afecb367f032d93F642f64180aa3", -}); -``` - -Connect to the indexer. -The `chainId` value is for a sanity check, to ensure we are connecting to the correct indexer. -The `address` parameter is the address of the `World` contract whose information we are reading. -The same indexer can read from multiple `World` objects, as long as they are on the same blockchain. - -```ts -console.log(`Block number: ${result.blockNumber}`); -``` - -The `result` has two fields: - -- `blockNumber`, the data is correct as of that block (it could change later). -- `tables`, the tables of the `World` we are reading. - -```ts -console.log(`Tables: ${result.tables.map((t) => t.name)}`); -``` - -Use [`map`](https://www.tutorialspoint.com/typescript/typescript_array_map.htm) to get the names of the tables. - -```ts -const counterTable = result.tables.filter((t) => t.name == "Counter")[0]; -``` - -Use [`filter`](https://www.tutorialspoint.com/typescript/typescript_array_filter.htm) to get the `Counter` table, assuming there is at least one. - -```ts -console.log("Information about Counter"); -console.log(counterTable); -``` - -Log the information we have about the table. -It contains these fields: - -| Field | Meaning | -| ---------------------- | ------------------------------------------------------------------------------------- | -| id | Iinternal ID for uniqueness in the context of SQL | -| address | `World` address | -| tableId | `bytes32` hex encoded table ID (a concat of `bytes16(namespace)` and `bytes16(name)`) | -| namespace | Table's namespace | -| name | Table's name | -| keySchema | Schema of the table key | -| valueSchema | Schema of the table value | -| lastUpdatedBlockNumber | Block number for which the data is correct | -| records | Actual table data | - -```ts -console.log(`The actual value: ${counterTable.records[0].value.value}`); -``` - -The actual value in `Counter`, which is in the first (and only) record. -Every record has two fields: - -- `key` -- `value` - -Both `key` and `value` can have multiple fields inside them, one for each field in the appropriate schema. -In the case of `Counter` there is nothing in the `key`, but `value` has a single field, also called `value`. -So the counter value is `value.value` of the record. - -### Using HTTP - - -

- The parameters, especially the URL, may change as we adjust our tRPC configuration. - - If you can use the TypeScript methods, those are more stable. -

- - -} -/> - -The indexer is built on top of [tRPC](https://trpc.io/), which is primarily designed to be used between [TypeScript](https://www.typescriptlang.org/) programs. -As a result, the data format is more complicated than REST APIs. - -For queries, the indexer expects to get two parameters in a [query string](https://en.wikipedia.org/wiki/Query_string): - -- `batch`, the batch identifier. - This parameter can contain any string, as long as it is present. - -- `input`, the actual query. - This query is a [URLEncoded](https://en.wikipedia.org/wiki/Query_string#URL_encoding) [JSON](https://en.wikipedia.org/wiki/JSON) object. - To play with possible object values, you can use [this online calculator](https://www.urlencoder.org/). - - For example, the input object used in the sanity check above is: - - ```json - { - "0": { - "json": { - "chainId": 31337, - "address": "0x5FbDB2315678afecb367f032d93F642f64180aa3" - } - } - } - ``` - - An HTTP request can include multiple queries, so the query object includes a numbered key (or numbered keys) with the actual queries. - Also, in the future other formats than JSON might be supported, so each query specifies what format it uses. - -The one type of query currently supported is `findAll`. It gives you all the information stored on the specific world in the indexer. -The query is available at path `/findAll` and have two parameters: - -- `chainId`: The chain ID of the blockchain to query. -- `address`: The address of the `World`. - Based on the paramters in the development environment, `pnpm dev` uses the address in the sanity check. - -To understand the result, it is easiest to look at it inside Node. - -1. Read the result and enter `node`: - - ```sh copy - curl 'http://localhost:3001/findAll?batch=wd&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' > res.json - node - ``` - -1. Read the result into a JavaScript object. - - ```javascript copy - res = JSON.parse(fs.readFileSync("res.json")); - ``` - -1. `res` is actually a list of results, which only has one item because we only had one key in the query. - This item has one key, `record`, whose value also has only one key, `data`. - Run these commands to verify the statements above and then cut `res` to the actual result. - - ```javascript copy - res.length; - Object.keys(res[0]); - Object.keys(res[0].result); - res = res[0].result.data; - ``` - -1. Inside the result there are two fields: - - - `meta`, the [metadata](https://en.wikipedia.org/wiki/Metadata). - In the case of JSON the field names are part of the data, so this primarily contains data types. - - `json`, the data read from the `World`. - This field itself is divided into two values: - - `blockNumber`, the block number in which these query results apply. - - `tables`, a list of the tables in the `World` and their data. - - You can use [the `filter` function](https://www.w3schools.com/jsref/jsref_filter.asp) to get a specific table. - In this case, we are getting the `Counter` table. - - ```javascript copy - counterTable = res.json.tables.filter((t) => t.name === "Counter")[0]; - ``` - -1. Finally, to get the actual record with the counter value, use: - - ```javascript copy - counterTable.records[0].value.value; - ``` - -## Debugging - -[See here](/reference#mud-server-components). - -## Indexing other blockchains - -To index a different blockchain, you specify these environment variables: - -| Variable | Value | -| ----------------------- | -------------------------------------------------------------------------------------------------------- | -| RPC_HTTP_URL | The URL to access the blockchain using HTTP | -| RPC_WS_URL | The URL to access the blockchain using WebSocket | -| START_BLOCK1 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | -| SQLITE_FILENAME | `.db` | - -(1) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed. - -After you do that, start the indexer with this command: - -```sh copy -pnpm start:sqlite -``` - -Note that in your queries you need to specify the correct chainId for the chain you are indexing. - -## Using the indexer with PostgreSQL - -To run the indexer with [PostgreSQL](https://www.postgresql.org/), follow these steps: - -1. Download and install [PostgreSQL](https://www.postgresql.org/download/). - -1. To use the default setting (local PostgreSQL, using the `postgres` database, indexing the local `anvil` chain), run: - - ```sh copy - pnpm start:postgres:local - ``` - - Otherwise, set these environment variables: - - | Variable | Value | - | ------------------------ | -------------------------------------------------------------------------------------------------------- | - | CHAIN_ID | ChainID for the chain you index | - | RPC_HTTP_URL1 | The URL to access the blockchain using HTTP | - | RPC_WS_URL1 | The URL to access the blockchain using WebSocket | - | START_BLOCK2 | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | - | DATABASE_URL | URL for the database, of the form `postgres:///` | - - (1) Not necessary in the case of a Lattice testnet. - - (2) This value is optional, but highly recommended in chains where the world was deployed after a large number of blocks had already passed for performance reasons. - - And then run: - - ```sh copy - pnpm start:postgres - ``` - -### Reading information from PostgreSQL - -You can now use [`psql`](https://www.postgresql.org/docs/current/app-psql.html) to view the indexer information. - -Note: This example uses [the template](http://localhost:3000/tutorials/minimal). -If you are indexing a different `World`, change the world address and use the name of a table you have rather than `Counter`. - -1. Get the list of schemas to verify `__mud_internal` is there. - - ```sql copy - \dn - ``` - -1. Get the list of relations in the schema. - - ```sql copy - \dt __mud_internal.* - ``` - - You should get two tables, `chain` and `tables`. - -1. Get the list of chains being indexed. - - ```sql copy - SELECT * FROM __mud_internal.chain; - ``` - -1. The `tables` table is a lot bigger, so first get the list of columns within the table. - - ```sql copy - \d+ __mud_internal.tables - ``` - -1. Get the list of MUD table names. - - ```sql copy - SELECT name FROM __mud_internal.tables; - ``` - -1. Get the key and value schemas for `Counter`. - - ```sql copy - SELECT key_schema, value_schema FROM __mud_internal.tables WHERE name='Counter'; - ``` - -1. The schema for a world is `
__`. - Get the list of tables for a specific world. - - ```sql copy - \dt "0x5FbDB2315678afecb367f032d93F642f64180aa3__".* - ``` - -1. Specify the world schema is the default. - - ```sql copy - SET search_path TO "0x5FbDB2315678afecb367f032d93F642f64180aa3__"; - ``` - -1. Get the value of the counter. - - ```sql copy - SELECT * FROM "Counter"; - ``` - -### Deleting the PostgreSQL data - -PostgreSQL is a persistent data storage. -This means that if you restart the blockchain without manually deleting the data, your indexer will use old data and provide incorrect results. - -To avoid this, enter `psql` and run this command: - -```sql copy -DROP OWNED BY ; -``` From 12bfcfad20e30cac843ebf9f4c46713bac4e79a9 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Tue, 17 Oct 2023 14:30:27 -0500 Subject: [PATCH 3/8] docs: add indexer to the new docs --- next-docs/pages/services/indexer.mdx | 32 +++++++++++++++++++++------- 1 file changed, 24 insertions(+), 8 deletions(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index 5e7c6747db..fba9f294f3 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -33,19 +33,25 @@ These environment variables need to be provided to the indexer to work: To run the indexer directly on your computer: -1. Create a `.env` file with the environment variables. - For example, if you are running the template with `pnpm dev` locally, you might use this file: +1. Build the MUD repository, [as per the directions](/contribute). - ```bash filename=".env" - RPC_HTTP_URL=http://localhost:8545 - ``` - -1. Install the indexer +1. Start a `World` to index. + An easy way to do this is to [use a TypeScript template](/templates/typescript/getting-started) in a separate command line window. - ```bash +1. Run the indexer from the build MUD repository. + For example, to start the indexer with SQLite, use these commands. + ```bash copy + cd packages/store-indexer + export RPC_HTTP_URL=http://localhost:8545 + pnpm start:sqlite ``` + Alternatively, you can start the indexer for the local blockchain using `pnpm start:sqlite:local`, in which case you don't need to specify `RPC_HTTP_URL`. + +**Note:** The `anvil.db` is persistent if you stop and restart the indexer. +If that is not the desired behavior (for example, because you restarted the blockchain itself), delete it before starting the indexer. + ### Docker The indexer Docker image is available [on github](https://github.com/latticexyz/mud/pkgs/container/store-indexer). @@ -150,3 +156,13 @@ The command to start the indexer in PostgreSQL mode is `pnpm start:postgres`. ```sql copy DROP OWNED BY root; ``` + +### Verification + +To verify that the indexer runs correctly ensure you have [`jq`](https://jqlang.github.io/jq/) installed and run + +```bash copy +curl 'http://localhost:3001/trpc/findAll?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' | jq +``` + +## Resetting the database From eff3204a1dacb6dc1b8ac39841333b7954380269 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Tue, 17 Oct 2023 14:31:25 -0500 Subject: [PATCH 4/8] Clarified ETH_RPC_URL --- next-docs/pages/services/indexer.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index fba9f294f3..98fa284327 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -20,13 +20,13 @@ This can also quickly accelerate the need to run a set of dedicated nodes just t These environment variables need to be provided to the indexer to work: -| Type | Variable | Meaning | Sample value (using `anvil` running on the host) | -| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------ | -| Needed | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 | +| Type | Variable | Meaning | Sample value (using `anvil` running on the host) | +| ----------------------------- | --------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | +| Needed | RPC_HTTP_URL | The URL to access the blockchain using HTTP | http://host.docker.internal:8545 (when running in Docker) | | Optional | RPC_WS_URL | The URL to access the blockchain using WebSocket | -| Optional | START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | 1 | -| Optional | DEBUG=mud:\* | Turn on debugging | | -| Optional, only for SQLite | SQLITE_FILENAME | Name of database | `anvil.db` | +| Optional | START_BLOCK | The block to start indexing from. The block in which the `World` contract was deployed is a good choice. | 1 | +| Optional | DEBUG=mud:\* | Turn on debugging | | +| Optional, only for SQLite | SQLITE_FILENAME | Name of database | `anvil.db` | | Optional, only for PostgreSQL | DATABASE_URL | URL for the database, of the form `postgres:///` | ### Running directly From 5ab67ae98a2963bc56a8692a7fd5b96c09abb2a9 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Fri, 20 Oct 2023 12:04:47 -0500 Subject: [PATCH 5/8] Apply suggestions from code review Co-authored-by: alvarius --- next-docs/pages/services/indexer.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index 98fa284327..121c2ad393 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -1,12 +1,11 @@ -import WarningBox from "../../components/WarningBox"; # Indexer -The MUD Indexer is an off-chain indexer for on-chain applications built with MUD. +The MUD Indexer is an offchain indexer for onchain applications built with MUD. ## Why an off-chain indexer? -Reads with on-chain apps can be tricky. +Reads with onchain apps can be tricky. What does it mean to be able to query the Ethereum network? Technically, given a node with a fully synced state, we can explore just about everything using the EVM, but the “exploring” would be looking at raw storage slots for accounts corresponding to smart contracts. A way around this exists by providing view functions on smart contracts: these effectively are just wrappers around raw storage and expose a more friendly API. From 313390b3bbfb8992ac5cd65e0e56780e9000a62b Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Fri, 20 Oct 2023 14:34:51 -0500 Subject: [PATCH 6/8] added resetting the database --- next-docs/pages/services/indexer.mdx | 31 +++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index 121c2ad393..ff9745dac4 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -1,9 +1,8 @@ - # Indexer The MUD Indexer is an offchain indexer for onchain applications built with MUD. -## Why an off-chain indexer? +## Why an offchain indexer? Reads with onchain apps can be tricky. What does it mean to be able to query the Ethereum network? @@ -11,9 +10,11 @@ Technically, given a node with a fully synced state, we can explore just about e A way around this exists by providing view functions on smart contracts: these effectively are just wrappers around raw storage and expose a more friendly API. Instead of having to figure out where the balances for an account are stored in the storage tree, we now can call a function that does the lookup via Solidity via an RPC endpoint. -The issue with view functions is that complexity in having to generate and call these to get the “full picture” of the state from the chain explodes pretty quickly. -For a web app client wanting to present the user with an up-to-date view of the on-chain state, the task of calling the different view functions becomes unnecessarily hard very fast. -This can also quickly accelerate the need to run a set of dedicated nodes just to be able to service the needed number of RPC requests. +The issue with view functions is that for any sophisticated application the calls needed to get the “full picture” of the state from the chain are very complex. +Servicing so many view function calls also creates the need to run a set of dedicated nodes instead of relying on a third-party provider's free tier. + +The MUD indexer solves this problem by listening to the MUD store events to automatically replicate the entire onchain state in a relational database. +Having such a database allows clients to quickly and efficiently query the onchain data. ## Installation @@ -164,4 +165,24 @@ To verify that the indexer runs correctly ensure you have [`jq`](https://jqlang. curl 'http://localhost:3001/trpc/findAll?batch=1&input=%7B%220%22%3A%7B%22json%22%3A%7B%22chainId%22%3A31337%2C%22address%22%3A%220x5FbDB2315678afecb367f032d93F642f64180aa3%22%7D%7D%7D' | jq ``` +The result should be nicely formatted (and long) JSON output with all the data stored in the `World`. + ## Resetting the database + +When you restart the blockchain you _have_ to reset the database, otherwise you'll get a mix of old and new information. +Here are the directions how to do it. + +### SQLite + +Delete the database file (by default `anvil.db`). + +### PostgreSQL + +Run `psql` and run this command: + +```sql +DROP OWNED BY ; +``` + +- If you are running the indexer locally, the owner of the tablespaces is your user name. +- If you are using Docker, the owner of the tablespaces is `root`. From 56e69ec810dc6e58183831fb8a3c3d74162d7381 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Mon, 23 Oct 2023 08:47:16 -0500 Subject: [PATCH 7/8] Update next-docs/pages/services/indexer.mdx Co-authored-by: alvarius --- next-docs/pages/services/indexer.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index ff9745dac4..cdaf34ef02 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -133,7 +133,7 @@ The command to start the indexer in PostgreSQL mode is `pnpm start:postgres`. CREATE ROLE root SUPERUSER LOGIN; ``` - **Note:** This is assuming a database that is isolated from the Internet and only used by trusted entities. + **Note:** This is assuming a database that is isolated from the internet and only used by trusted entities. In a production system you will use at least a password as authentication, and limit the user's authority. 1. Start the docker container. From 40b70c2c4c6ff18dca891aeb3496db56160418c6 Mon Sep 17 00:00:00 2001 From: Ori Pomerantz Date: Mon, 23 Oct 2023 08:48:53 -0500 Subject: [PATCH 8/8] Update indexer.mdx --- next-docs/pages/services/indexer.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/next-docs/pages/services/indexer.mdx b/next-docs/pages/services/indexer.mdx index cdaf34ef02..61c46dc2b8 100644 --- a/next-docs/pages/services/indexer.mdx +++ b/next-docs/pages/services/indexer.mdx @@ -64,7 +64,7 @@ There are several ways to provide the environment variables to `docker run`: You specify this after the `docker run`, but before the name of the image. - Both [Docker Compose](https://docs.docker.com/compose/) and [Kubernetes](https://kubernetes.io/) have their own mechanisms for starting docker containers with environment variables. -The easiest way to test the indexer is to [run the template as a world]() in a separate command-line window. +The easiest way to test the indexer is to [run the template as a world](/templates/typescript/getting-started) in a separate command-line window. #### SQLite