7.3.0 release docs

docs/1.getting-started/3.config.md

@@ -5,29 +5,31 @@ description: "Developing a DipDup indexer begins with creating a YAML config fil
 
 # Creating config
 
-Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expect it to be located in the project root and named `dipdup.yaml`. However, you can provide any path with a `-c` CLI option.
+Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expects it to be located in the project root and named `dipdup.yaml`. However, you can provide any path with a `-c` CLI option.
 
 ## Structure
 
 Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is **Header**.
 
-|                  |                                                 |                                                                       |
-| ---------------- | ----------------------------------------------- | --------------------------------------------------------------------- |
-| **Header**       | [spec_version](../9.config/14.spec_version.md)* | DipDup project specification; currently "2.0"                         |
-|                  | [package](../9.config/11.package.md)*           | Python package name                                                   |
-| **Inventory**    | [database](../9.config/4.database.md)           | Database configuration                                                |
-|                  | [contracts](../9.config/2.contracts.md)         | Contracts that need to be indexed                                     |
-|                  | [datasources](../9.config/5.datasources.md)     | Datasources to gather information from                                |
-| **Indexes**      | [indexes](../9.config/8.indexes.md)             | Index definitions                                                     |
-|                  | [templates](../9.config/15.templates.md)        | Also index definitions, but with placeholders that make them reusable |
-| **Hooks**        | [hooks](../9.config/7.hooks.md)                 | Callbacks to run manually or by schedule                              |
-|                  | [jobs](../9.config/9.jobs.md)                   | Schedules for hooks                                                   |
-| **Integrations** | [hasura](../9.config/6.hasura.md)               | Hasura GraphQL Engine configuration                                   |
-|                  | [sentry](../9.config/13.sentry.md)              | Sentry configuration                                                  |
-|                  | [prometheus](../9.config/12.prometheus.md)      | Prometheus configuration                                              |
-| **Miscellaneous**| [advanced](../9.config/1.advanced.md)           | Tunables that affect framework behavior                               |
-|                  | [custom](../9.config/3.custom.md)               | Mapping of user-defined values; neither typed nor validated           |
-|                  | [logging](../9.config/10.logging.md)            | Configure logging verbosity                                           |
+See [Config reference guide](../7.references/2.config.md) for the full list of available options.
+
+|                   |                 |                                                                       |
+| ----------------- | --------------- | --------------------------------------------------------------------- |
+| **Header**        | `spec_version`* | DipDup project specification; currently "2.0"                         |
+|                   | `package`*      | Python package name                                                   |
+| **Inventory**     | `database`      | Database configuration                                                |
+|                   | `contracts`     | Contracts that need to be indexed                                     |
+|                   | `datasources`   | Datasources to gather information from                                |
+| **Indexes**       | `indexes`       | Index definitions                                                     |
+|                   | `templates`     | Also index definitions, but with placeholders that make them reusable |
+| **Hooks**         | `hooks`         | Callbacks to run manually or by schedule                              |
+|                   | `jobs`          | Schedules for hooks                                                   |
+| **Integrations**  | `hasura`        | Hasura GraphQL Engine configuration                                   |
+|                   | `sentry`        | Sentry configuration                                                  |
+|                   | `prometheus`    | Prometheus configuration                                              |
+| **Miscellaneous** | `advanced`      | Tunables that affect framework behavior                               |
+|                   | `custom`        | Mapping of user-defined values; neither typed nor validated           |
+|                   | `logging`       | Configure logging verbosity                                           |
 
 ## Merging multiple files
 
@@ -80,4 +82,4 @@ contracts:
     typename: quipu_fa12
 ```
 
-If multiple contracts you index have the same interface but different code, see [F.A.Q.](../12.faq.md) learn how to avoid conflicts.
+If multiple contracts you index have the same interface but different code, see [F.A.Q.](../12.faq.md) to learn how to avoid conflicts.

docs/1.getting-started/6.datasources.md

@@ -9,16 +9,16 @@ Datasources are DipDup connectors to various APIs. They are defined in config an
 
 Index datasources, ones that can be attached to a specific index, are prefixed with blockchain name, e.g. `tezos.tzkt` or `evm.subsquid`.
 
-| kind                                      | blockchain     | description                     |
-| ----------------------------------------- | -------------- | ------------------------------- |
-| [abi.etherscan](2.abi_etherscan.md)       | EVM-compatible | Provides ABIs for EVM contracts |
-| [coinbase](3.coinbase.md)                 | any            | Coinbase price feed             |
-| [evm.node](4.evm_node.md)                 | EVM-compatible | Ethereum node                   |
-| [evm.subsquid](5.evm_subsquid.md)         | EVM-compatible | Subsquid Archives node                   |
-| [http](6.http.md)                         | any            | Generic HTTP API                        |
-| [ipfs](7.ipfs.md)                         | any            | IPFS gateway                            |
-| [tezos.tzkt](8.tezos_tzkt.md)             | Tezos          | TzKT API                        |
-| [tezos.tzip_metadata](9.tzip_metadata.md) | Tezos          | TZIP-16 metadata                |
+| kind                                                       | blockchain     | description                     |
+| ---------------------------------------------------------- | -------------- | ------------------------------- |
+| [abi.etherscan](../3.datasources/1.abi_etherscan.md)       | EVM-compatible | Provides ABIs for EVM contracts |
+| [coinbase](../3.datasources/2.coinbase.md)                 | any            | Coinbase price feed             |
+| [evm.node](../3.datasources/3.evm_node.md)                 | EVM-compatible | Ethereum node                   |
+| [evm.subsquid](../3.datasources/4.evm_subsquid.md)         | EVM-compatible | Subsquid Archives node          |
+| [http](../3.datasources/5.http.md)                         | any            | Generic HTTP API                |
+| [ipfs](../3.datasources/6.ipfs.md)                         | any            | IPFS gateway                    |
+| [tezos.tzkt](../3.datasources/7.tezos_tzkt.md)             | Tezos          | TzKT API                        |
+| [tezos.tzip_metadata](../3.datasources/8.tzip_metadata.md) | Tezos          | TZIP-16 metadata                |
 
 ## Connection settings
 

docs/1.getting-started/7.indexes.md

@@ -42,7 +42,7 @@ indexes:
       datasource: tzkt_mainnet
 ```
 
-You can also spawn indexes from templates in runtime; see [Variables](../1.getting-started/7.env-variables.md) page.
+You can also spawn indexes from templates in runtime; see [Spawning in runtime](../1.getting-started/7.indexes.md#spawning-in-runtime).
 
 ## Limiting scope
 

docs/5.advanced/3.sql.md

@@ -17,8 +17,8 @@ Several tables haing `dipdup_` prefix are created by DipDup automatically and ar
 | `dipdup_contract`          | Info about contracts used by all indexes, including ones added in runtime.                                                                |
 | `dipdup_model_update`      | Service table to store model diffs for database rollback. Configured by `advanced.rollback_depth`                                         |
 | `dipdup_meta`              | Arbitrary key-value storage for DipDup internal use. Survives reindexing. You can use it too, but don't touch keys with `dipdup_` prefix. |
-| `dipdup_contract_metadata` | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
-| `dipdup_token_metadata`    | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
+| `dipdup_contract_metadata` | See [Metadata interface](4.metadata-interface.md)                                                                               |
+| `dipdup_token_metadata`    | See [Metadata interface](4.metadata-interface.md)                                                                               |
 
 See [`dipdup.models` module](https://github.com/dipdup-io/dipdup/blob/next/src/dipdup/models/__init__.py) for exact table definitions.
 

docs/5.advanced/4.metadata-interface.md

@@ -99,7 +99,7 @@ query MyQuery {
 }
 ```
 
-DipDup handles table management for you and exposes a [context-level helper](../advanced/context.md).
+DipDup handles table management for you and exposes a helper.
 
 Tezos Domains example:
 

docs/6.deployment/4.prometheus.md

@@ -33,4 +33,4 @@ Also, DipDup exposes the following metrics for compatibility with Subsquid Cloud
 | `sqd_processor_chain_height`               | Current chain height as reported by Subsquid Archives      |
 | `sqd_processor_archive_http_errors_in_row` | Number of consecutive failed requests to Subsquid Archives |
 
-If you need more complex metrics, consider querying [Internal tables](../5.advanced/3.internal-tables.md).
+If you need more complex metrics, consider querying [Internal tables](../5.advanced/3.sql.md#internal-tables).

docs/7.references/2.config.md

@@ -43,13 +43,12 @@ description: "Config file reference"
 <li><p><strong>reindex</strong> (<em>dict</em><em>[</em><a class="reference internal" href="#dipdup.models.ReindexingReason" title="dipdup.models.ReindexingReason"><em>ReindexingReason</em></a><em>, </em><a class="reference internal" href="#dipdup.models.ReindexingAction" title="dipdup.models.ReindexingAction"><em>ReindexingAction</em></a><em>]</em>) – Mapping of reindexing reasons and actions DipDup performs</p></li>
 <li><p><strong>scheduler</strong> (<em>dict</em><em>[</em><em>str</em><em>, </em><em>Any</em><em>] </em><em>| </em><em>None</em>) – <cite>apscheduler</cite> scheduler config</p></li>
 <li><p><strong>postpone_jobs</strong> (<em>bool</em>) – Do not start job scheduler until all indexes are in realtime state</p></li>
-<li><p><strong>early_realtime</strong> (<em>bool</em>) – Establish realtime connection immediately after startup</p></li>
+<li><p><strong>early_realtime</strong> (<em>bool</em>) – Spawn realtime datasources immediately after startup</p></li>
 <li><p><strong>skip_version_check</strong> (<em>bool</em>) – Do not check for new DipDup versions on startup</p></li>
 <li><p><strong>rollback_depth</strong> (<em>int</em><em> | </em><em>None</em>) – A number of levels to keep for rollback</p></li>
 <li><p><strong>decimal_precision</strong> (<em>int</em><em> | </em><em>None</em>) – Overwrite precision if it’s not guessed correctly based on project models.</p></li>
 <li><p><strong>unsafe_sqlite</strong> (<em>bool</em>) – Disable journaling and data integrity checks. Use only for testing.</p></li>
-<li><p><strong>metrics</strong> – off/basic/advanced based on how much performance metrics you want to collect</p></li>
-<li><p><strong>alt_operation_matcher</strong> (<em>bool</em>) – Use different algorithm to match operations (dev only)</p></li>
+<li><p><strong>alt_operation_matcher</strong> (<em>bool</em>) – Use different algorithm to match Tezos operations (dev only)</p></li>
 </ul>
 </dd>
 </dl>

docs/8.examples/1.demos.md

@@ -33,7 +33,7 @@ Covers all available operations of Quipuswap DEX contracts: trades, transfers, m
 
 source: [demo_factories](https://github.com/dipdup-io/dipdup/tree/next/src/demo_factories)
 
-Homebase enables users to create DAO contracts. In this example indexes are spawned in runtime ([spawning in runtime](../1.getting-started/7.indexes.md?Spawning-in-runtime)) for all contracts having the same script.
+Homebase enables users to create DAO contracts. In this example, indexes are spawned in runtime (see [Spawning in runtime](../1.getting-started/7.indexes.md#spawning-in-runtime)) for all contracts having the same script.
 
 ## Tezos Domains (`operation`)
 

docs/9.release-notes/1.v7.2.md

@@ -83,7 +83,7 @@ curl 127.0.0.1:46339/add_index -X POST -H "Content-Type: application/json" \
 curl 127.0.0.1:46339/performance
 ```
 
-See the [API reference](../7.references/5.api.md) for more details.
+See the [API reference](../7.references/4.api.md) for more details.
 
 ## Other improvements
 

docs/9.release-notes/3.v7.0.md

@@ -102,7 +102,7 @@ See the [Docker](../6.deployment/2.docker.md) page in the docs.
 
 Here are some other notable changes not covered above:
 
-- `dipdup_meta` internal table was added to the schema. You can store arbitrary JSON there and use it in your code, but don't touch records with `dipdup_` prefix. Survives reindexing. See the [Internal tables](../1.getting-started/6.internal-tables.md) page.
+- `dipdup_meta` internal table was added to the schema. You can store arbitrary JSON there and use it in your code, but don't touch records with `dipdup_` prefix. Survives reindexing. See the [Internal tables](../5.advanced/3.sql.md#internal-tables) page.
 - Multiple feature flags were added for experimental and rarely used features. See the [Feature flags](../5.advanced/2.feature-flags.md) page. `metadata_interface` flag was removed, now it's always enabled.
 - We no longer accept crash reports. Enabling them required adding `crash_reporting: True` to the config. Set up your own Sentry instance; it's easy!
 - Saved crash- and performance reports in the home directory can be viewed with new `report ls` and `report show` commands. If you want to open an issue and share this report, just drop us a message on GitHub or Discord.

docs/9.release-notes/4.v6.5.md

@@ -7,4 +7,4 @@ DipDup 6.5 is the current old-stable release powering dozens of APIs in producti
 
 - [Documentation](https://docs.dipdup.io)
 - [Changelog](https://docs.dipdup.io/CHANGELOG)
-- [7.0 Migration Guide](docs/release-notes/v7.0#migration-guide)
+- [7.0 Migration Guide](../9.release-notes/3.v7.0.md#migration-guide)

docs/9.release-notes/5.history.md

@@ -329,7 +329,7 @@ Pro tip: you can also enable notifications on the GitHub repo page with _👁 Wa
 
 ### Changelog
 
-See full 5.1.0 changelog [here](../CHANGELOG.md#510---2022-05-12).
+See the full 5.1.0 changelog [here](../../CHANGELOG.md).
 
 ## 5.0.0
 
@@ -720,7 +720,7 @@ There are several reasons that trigger reindexing:
 |`migration`|Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared.|
 |`rollback`|Reorg message received from TzKT, and can not be processed.|
 |`config_modified`|One of the index configs has been modified.|
-|`schema_modified`|Database schema has been modified. Try to avoid manual schema modifications in favor of [SQL hooks](../advanced/sql.md).|
+|`schema_modified`|Database schema has been modified. Try to avoid manual schema modifications in favor of [SQL hooks](../5.advanced/3.sql.md)|
 
 Now it is possible to configure desirable action on reindexing triggered by the specific reason.
 

pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "dipdup"
 description = "Modular framework for creating selective indexers and featureful backends for dapps"
-version = "7.2.2"
+version = "7.3.0"
 license = { text = "MIT" }
 authors = [
     { name = "Lev Gorodetskii", email = "[email protected]" },

scripts/check_links.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+import logging
+from pathlib import Path
+import re
+
+logging.basicConfig(level=logging.INFO, format='%(levelname)-8s %(message)s')
+
+MD_LINK_REGEX = r'\[.*\]\(([0-9a-zA-Z\.\-\_\/\#\:\/\=\?]*)\)'
+ANCHOR_REGEX = r'\#\#* [\w ]*'
+
+files, links, http_links, bad_links, bad_anchors = 0, 0, 0, 0, 0
+
+for path in Path('docs').rglob('*.md'):
+    logging.info('checking file `%s`', path)
+    files += 1
+    data = path.read_text()
+    for match in re.finditer(MD_LINK_REGEX, data):
+        links += 1
+        link = match.group(1)
+        if link.startswith('http'):
+            http_links += 1
+            continue
+
+        link, anchor = link.split('#') if '#' in link else (link, None)
+
+        full_path = path.parent.joinpath(link)
+        if not full_path.exists():
+            logging.error('broken link: `%s`', full_path)
+            bad_links += 1
+            continue
+
+        if anchor:
+            target = full_path.read_text() if link else data
+
+            for match in re.finditer(ANCHOR_REGEX, target):
+                header = match.group(0).lower().replace(' ', '-').strip('#-')
+                if header == anchor.lower():
+                    break
+            else:
+                logging.error('broken anchor: `%s#%s`', link, anchor)
+                bad_anchors += 1
+                continue
+
+logging.info('_' * 80)
+logging.info('checked %d files and %d links:', files, links)
+logging.info('%d URLs, %d bad links, %d bad anchors', http_links, bad_links, bad_anchors)

src/demo_auction/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_auction"
 version = "0.0.1"

src/demo_big_maps/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_big_maps"
 version = "0.0.1"

src/demo_blank/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_blank"
 version = "0.0.1"

src/demo_dao/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_dao"
 version = "0.0.1"

src/demo_dex/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_dex"
 version = "0.0.1"

src/demo_domains/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_domains"
 version = "0.0.1"

src/demo_etherlink/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_etherlink"
 version = "0.0.1"

src/demo_events/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_events"
 version = "0.0.1"

src/demo_evm_events/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_evm_events"
 version = "0.0.1"

src/demo_factories/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_factories"
 version = "0.0.1"

src/demo_head/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_head"
 version = "0.0.1"

src/demo_nft_marketplace/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_nft_marketplace"
 version = "0.0.1"

src/demo_raw/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_raw"
 version = "0.0.1"

src/demo_token/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_token"
 version = "0.0.1"

src/demo_token_balances/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_token_balances"
 version = "0.0.1"

src/demo_token_transfers/pyproject.toml

@@ -1,4 +1,4 @@
-# Generated by DipDup 7.2.2+editable
+# Generated by DipDup 7.3.0+editable
 [project]
 name = "demo_token_transfers"
 version = "0.0.1"