Merge branch 'next' into feat/config-env-compose

dipdup-io · Oct 18, 2023 · 2030bbb · 2030bbb
2 parents 8e625b2 + f1ad9cb
commit 2030bbb
Show file tree

Hide file tree

Showing 39 changed files with 632 additions and 422 deletions.
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -17,7 +17,7 @@ on:
 
 env:
   FRONTEND_BRANCH: master
-  GH_TOKEN: ${{ secrets.API_TOKEN_EXT }}
+  GITHUB_TOKEN: ${{ secrets.DOCS_GITHUB_TOKEN }}
 
 jobs:
   docs:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -10,6 +10,37 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 
 - cli: Added `--unsafe` and `--compose` flags to `config env` command.
 
+## [7.0.2] - 2023-10-10
+
+### Added
+
+- database: Added `dipdup_wipe` and `dipdup_approve` SQL functions to the schema.
+
+### Fixed
+
+- cli: Fixed `schema wipe` command for SQLite databases.
+- tezos.tzkt: Fixed regression in `get_transactions` method pagination.
+
+## [6.5.13] - 2023-10-10
+
+### Fixed
+
+- tzkt: Fixed regression in `get_transactions` method pagination.
+
+## [7.0.1] - 2023-09-30
+
+### Added
+
+- env: Added `DIPDUP_DEBUG` environment variable to enable debug logging.
+
+### Fixed
+
+- cli: Use correct data path with timescaledb-ha Docker image.
+- demos: Fixed decimal overflow in `demo_uniswap` project.
+- evm.node: Fixed incorrect log request parameters.
+- evm.subsquid.events: Fixed issue with determining the last level when syncing with node.
+- hasura: Increated retry count for initial connection (healthcheck).
+
 ## [7.0.0] - 2023-09-25
 
 ### Fixed
@@ -1185,7 +1216,8 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.0...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.1...HEAD
+[7.0.1]: https://github.com/dipdup-io/dipdup/compare/7.0.0...7.0.1
 [7.0.0]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc5...7.0.0
 [7.0.0rc5]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc4...7.0.0rc5
 [6.5.11]: https://github.com/dipdup-io/dipdup/compare/6.5.10...6.5.11

diff --git a/docs/5.advanced/1.reindexing.md b/docs/5.advanced/1.reindexing.md
@@ -7,13 +7,13 @@ description: "In some cases, DipDup can't proceed with indexing without a full w
 
 In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing:
 
-| reason            | description                                                                                                               |
-| ----------------- | ------------------------------------------------------------------------------------------------------------------------- |
-| `manual`          | Reindexing triggered manually from callback with `ctx.reindex`.                                                           |
-| `migration`       | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared. |
-| `rollback`        | Reorg message received from TzKT can not be processed.                                                                    |
-| `config_modified` | One of the index configs has been modified.                                                                               |
-| `schema_modified` | Database schema has been modified. Try to avoid manual schema modifications in favor of [sql](../5.advanced/6.sql.md).   |
+| reason            | description                                                                                                                    |
+| ----------------- | ------------------------------------------------------------------------------------------------------------------------------ |
+| `manual`          | Reindexing triggered manually from callback with `ctx.reindex`.                                                                |
+| `migration`       | Applied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared.      |
+| `rollback`        | Reorg message received from datasource and can not be processed.                                                               |
+| `config_modified` | One of the index configs has been modified.                                                                                    |
+| `schema_modified` | Database schema has been modified. Try to avoid manual schema modifications in favor of [SQL scripts](../5.advanced/3.sql.md). |
 
 It is possible to configure desirable action on reindexing triggered by a specific reason.
 

diff --git a/docs/5.advanced/3.internal-tables.md → docs/5.advanced/3.sql.md b/docs/5.advanced/3.internal-tables.md → docs/5.advanced/3.sql.md
@@ -1,11 +1,13 @@
 ---
-title: "Internal tables"
-description: "This page describes the internal tables used by DipDup. They are created automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging."
+title: "Advanced SQL"
+description: "Put your *.sql scripts to dipdup_indexer/sql. You can run these scripts from any callback with ctx.execute_sql('name'). If name is a directory, each script it contains will be executed."
 ---
 
-# Internal tables
+# Advanced SQL
 
-This page describes the internal tables used by DipDup. They are created automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging.
+## Internal tables
+
+Several tables haing `dipdup_` prefix are created by DipDup automatically and are not intended to be modified by the user. However, they can be useful for external monitoring and debugging.
 
 | table                      | description                                                                                                                               |
 |:-------------------------- |:----------------------------------------------------------------------------------------------------------------------------------------- |
@@ -15,8 +17,8 @@ This page describes the internal tables used by DipDup. They are created automat
 | `dipdup_contract`          | Info about contracts used by all indexes, including ones added in runtime.                                                                |
 | `dipdup_model_update`      | Service table to store model diffs for database rollback. Configured by `advanced.rollback_depth`                                         |
 | `dipdup_meta`              | Arbitrary key-value storage for DipDup internal use. Survives reindexing. You can use it too, but don't touch keys with `dipdup_` prefix. |
-| `dipdup_contract_metadata` | See Metadata interface page                                                                                                               |
-| `dipdup_token_metadata`    | See Metadata interface page                                                                                                               |
+| `dipdup_contract_metadata` | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
+| `dipdup_token_metadata`    | See [Metadata interface](/docs/advanced/metadata-interface)                                                                               |
 
 See [`dipdup.models` module](https://github.com/dipdup-io/dipdup/blob/next/src/dipdup/models/__init__.py) for exact table definitions.
 
@@ -32,3 +34,28 @@ SELECT name, status FROM dipdup_index;
 -- Get last reindex time
 SELECT created_at FROM dipdup_schema WHERE name = 'public';
 ```
+
+## Scripts
+
+Put your `*.sql` scripts to `{{ project.package }}/sql`. You can run these scripts from any callback with `ctx.execute_sql('name')`. If `name` is a directory, each script it contains will be executed.
+
+Scripts are executed without being wrapped with SQL transactions. It's generally a good idea to avoid touching table data in scripts.
+
+By default, an empty `sql/<hook_name>` directory is generated for every hook in config during init. Remove `ctx.execute_sql` call from hook callback to avoid executing them.
+
+```python
+# Execute all scripts in sql/my_hook directory
+await ctx.execute_sql('my_hook')
+
+# Execute a single script
+await ctx.execute_sql('my_hook/my_script.sql')
+```
+
+## Managing schema
+
+When using PostgreSQL as database engine you can use `dipdup_approve` and `dipdup_wipe` functions to manage schema state from SQL console if needed:
+
+```sql
+SELECT dipdup_approve('public');
+SELECT dipdup_wipe('public');
+```
diff --git a/docs/5.advanced/6.sql.md b/docs/5.advanced/6.sql.md
diff --git a/docs/6.deployment/1.database.md b/docs/6.deployment/1.database.md
@@ -24,3 +24,30 @@ Immune tables support for SQLite is experimental and requires `advanced.unsafe_s
 Latest PostgreSQL and TimescaleDB versions are recommended due to significant performance improvements in recent releases (see [Feature matrix](https://www.postgresql.org/about/featurematrix/) page).
 
 Usually it's okay to use different database engines for development and production, but be careful with SQL scripts and column types that can behave differently.
+
+## TimescaleDB
+
+TimescaleDB is a PostgreSQL extension that provides time-series data management capabilities. You can use it with DipDup as a drop-in replacement for PostgreSQL.
+
+You can choose `timescale/timescaledb` or `timescale/timescaledb-ha` Docker images when initializing DipDup project.
+
+::banner{type="warning"}
+Be careful! Unlike other PostgreSQL images `timescale/timescaledb-ha` uses `/home/postgres/pgdata/data` as a persistent volume.
+::
+
+Use SQL scripts in `on_reindex` directory to prepare the database. First, create a hypertable replacing primary key with composite one:
+
+```sql [sql/on_reindex/00_prepare_db.sql]
+{{ #include ../src/demo_uniswap/sql/on_reindex/00_prepare_db.sql }}
+```
+
+Now you can create a continuous aggregate on top of the hypertable:
+
+```sql [sql/on_reindex/01_create_mv_token_price.sql]
+{{ #include ../src/demo_uniswap/sql/on_reindex/01_create_mv_token_price.sql }}
+```
+
+For more information visit the official TimescaleDB documentation:
+
+- [Hypertables](https://docs.timescale.com/use-timescale/latest/hypertables/)
+- [Continuous aggregates](https://docs.timescale.com/use-timescale/continuous-aggregates/)
diff --git a/docs/9.release-notes/1.v7.0.md b/docs/9.release-notes/1.v7.0.md
@@ -1,19 +1,19 @@
 ---
 title: 7.0.0
-description: DipDup 7.0.0 release notes
+description: DipDup 7.0 release notes
 ---
 
-# Release Notes: 7.0.0
+# Release Notes: 7.0
 
 Welcome, developers! Today we introduce 7.0, the most significant major release for DipDup in terms of both changes and developer hours. The new framework architecture allows to easily integrate new blockchains and data sources. EVM support is the first step in this direction; more to come soon. Also we have focused on improving developer experience, so you can initialize, extend and maintain DipDup projects with minimal effort. Finally, updated docs and new demo projects won't let you get lost.
 
 Key highlights:
 
-- Support for EVM-compatible blockchains
-- Updated project package structure with configs and deployment recipes
-- Storage layer improvements; a path to DipDup ORM
+- Support for EVM-compatible blockchains.
+- Updated project package structure with configs and deployment recipes.
+- Storage layer improvements; first steps path to DipDup ORM.
 - A Python 3.11 environment, updated Docker images, and better performance.
-- New convenient CLI commands
+- New convenient CLI commands.
 
 Join our socials to discuss this release and ask any questions!
 
@@ -23,17 +23,17 @@ Join our socials to discuss this release and ask any questions!
 
 Now DipDup supports EVM-compatible blockchains in addition to Tezos. The new index allows you to process contract events from Ethereum and other EVM-compatible networks. DipDup uses historical data from [Subsquid Archives](https://www.subsquid.io/), real-time data from RPC nodes, and ABIs from [Etherscan](https://etherscan.io/). All you need is to define an index in your config and implement handlers for each event.
 
-We have two demo projects for EVM: a very basic USDt price indexer and a more complex one for Uniswap V3 protocol. Run the `dipdup new` command, choose "EVM" on the first page, then a template to use.
+We have two demo projects for EVM: a very basic USDt price indexer and a more complex one for Uniswap v3 protocol. Run the `dipdup new` command, choose "EVM" on the first page, then a template to use.
 
 ## Project package
 
-The project package structure was updated become more consistent and easier to extend. Every package now have a fixed structure with directories for code, configs, SQL, etc. It allows discovery of package contents and increases the overall readability.
+The project package structure was updated to become more consistent and easier to extend. Every package now have a fixed structure with directories for code, configs, SQL, etc. It allows discovery of package contents and increases the overall readability.
 
 The DipDup package now includes three new sections:
 
 - `models` section replaces `models.py` module as a source of DipDup ORM models. You can use any structure inside; models will be discovered automatically.
 - `configs` directory contains files to extend the root config with environment-specific settings like database connection or logging. Keep these settings separated from the root config to make it more readable and declarative.
-- `deploy` directory contains Dockerfiles, Compose stack definitions and other deployment recipes. Also, there are `.env.default` files for each config in the `configs` directory. Use them as a template to never miss or commit an environment variable again.
+- `deploy` directory contains Dockerfiles, Compose stack definitions and other deployment recipes. Also, there are `.env.default` files for each config in the `configs` directory. Use them as a template never to miss or commit an environment variable again.
 
 The new `dipdup package tree` command allows inspecting the package structure and ensuring that everything is in place.
 
@@ -68,7 +68,7 @@ curl -Lsf https://dipdup.io/install.py | python3
 dipdup new
 ```
 
-Package and config discovery was improved, so managing a project is a bit easier now. You can omit default the `dipdup.y[a]ml` filename. Also, a package can be a workdir now saving you from typing lengthy paths.
+Package and config discovery was improved, so managing a project is a bit easier now. You can omit default the `dipdup.y[a]ml` filename. Also, a package root can be a workdir now (optional; creates a magic symlink).
 
 ```shell [Terminal]
 dipdup -c . -c configs/dipdup.sqlite.yaml config export
@@ -82,7 +82,9 @@ dipdup self install --force --ref next
 dipdup self uninstall
 ```
 
-Starting 7.0 we use [PDM](https://pdm.fming.dev/) as a default package manager. It's a swiss-knife to deal with Python's packaging pain with great PEP compatibility and a lot of features. Also, it can run scripts from pyproject.toml as npm does. We have updated the default template to use PDM instead of Poetry and GNU Make and included some scripts.
+Starting 7.0 we use [PDM](https://pdm.fming.dev/) as a default package manager. It's a swiss-knife to deal with Python's packaging pain with great PEP compatibility and a lot of features. Also, it can run scripts from pyproject.toml as npm does. DipDup projects now use a single, well PEP-compliant pyproject.toml for everything code-related. Explore it to find useful scripts for development and deployment.
+
+Poetry and GNU Make were removed from the default template, but you can still use them if you like.
 
 See the [Installation](../1.getting-started/1.installation.md) page in the docs.
 
@@ -92,7 +94,7 @@ DipDup projects now run on Python 3.11. Performance improvements introduced in t
 
 We have improved pre-fetching and caching data during indexing to increase the indexing speed.
 
-Docker images are now based on Debian 12. They are simple, secure and easy to extend - just run pip as a default user. Alpine images are no longer published, but migration should be seamless.
+Docker images are now based on Debian 12. They are simple, secure and easy to extend - just run pip as a default user. Alpine images are no longer published due to the lack of support in one of libraries we depend on, but migration should be seamless.
 
 See the [Docker](../6.deployment/2.docker.md) page in the docs.
 
@@ -101,15 +103,16 @@ See the [Docker](../6.deployment/2.docker.md) page in the docs.
 Here are some other notable changes not covered above:
 
 - `dipdup_meta` internal table was added to the schema. You can store arbitrary JSON there and use it in your code, but don't touch records with `dipdup_` prefix. Survives reindexing. See the [Internal tables](../1.getting-started/6.internal-tables.md) page.
-- Multiple feature flags were added for experimental and rarely used features. `metadata_interface` and `crash_reporting` flags (both disabled by default) were removed from the config. See the [Feature flags](../5.advanced/2.feature-flags.md) page.
-- Saved crash- and performance reports in the home directory can be viewed with new `report ls` and `report show` commands. If you want to report an issue and share the report, just drop us a message on GitHub or Discord.
+- Multiple feature flags were added for experimental and rarely used features. See the [Feature flags](../5.advanced/2.feature-flags.md) page. `metadata_interface` flag was removed, now it's always enabled.
+- We no longer accept crash reports. Enabling them required adding `crash_reporting: True` to the config. Set up your own Sentry instance; it's easy!
+- Saved crash- and performance reports in the home directory can be viewed with new `report ls` and `report show` commands. If you want to open an issue and share this report, just drop us a message on GitHub or Discord.
 - You can use long syntax to configure logging, a mapping of logger name and loglevel. See the [Logging](../6.deployment/5.logging.md) page.
-- YAML files in templates and examples use `.yaml` extension instead of `.yml` as recommended by the YAML spec, but you can use any.
+- YAML files in templates and examples use `.yaml` extension instead of `.yml` as recommended by the YAML and Compose specs, but you can use any.
 - `report` command has been renamed to `report ls` for consistency with other command groups.
 
 ## Future of DipDup 6.5
 
-The previous version of the framework is powering dozens of APIs in production. We want to give those projects enough time for migration or to just wait a bit while the current branch is being ironed out. Given that, _DipDup 6.5 will be supported until March 2024_. We will continue to release bugfixes and security updates until that date. You can find the old documentation [here](https://docs.dipdup.io/), but some pages may be outdated. If you've found such page or other issue, please drop us a message.
+The previous version of the framework is powering dozens of APIs in production. We want to give those projects enough time for migration or to wait a bit while the current branch is being ironed out. Given that, **DipDup 6.5 will be supported until March 2024**. We will continue to release bugfixes and security updates until that date. You can find the old documentation [here](https://docs.dipdup.io/), but some pages may be outdated. If you've found such a page or other issue, please drop us a message.
 
 Of course, we encourage you migrating to 7.0 soon to explore all the cool stuff in this release.
 

diff --git a/docs/9.release-notes/2.v6.2.md b/docs/9.release-notes/2.v6.2.md
@@ -1,6 +1,6 @@
 ---
 title: 6.2.0
-description: DipDup v6.2 release notes
+description: DipDup 6.2 release notes
 ---
 
 ## What's New
-Original file line number
+Diff line change
@@ Expand Up / @@ -17,7 +17,7 @@ on: @@
     env:
       FRONTEND_BRANCH: master
-      GH_TOKEN: ${{ secrets.API_TOKEN_EXT }}
+      GITHUB_TOKEN: ${{ secrets.DOCS_GITHUB_TOKEN }}
     jobs:
       docs:
@@ Expand Down @@