7.0.0rc4 release (#806)

Co-authored-by: Vladimir Bobrikov <[email protected]>

.github/workflows/release.yml

@@ -46,7 +46,7 @@ jobs:
         with:
           images: |
             dipdup/dipdup
-            ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}/dipdup
+            ${{ env.DOCKER_REGISTRY }}/${{ env.DOCKER_IMAGE_NAME }}
           flavor: |
             latest=false
           tags: |
@@ -79,6 +79,7 @@ jobs:
       - name: Publish package on PyPi
         run: pdm publish --password ${{secrets.PYPI_TOKEN}}
 
+      # FIXME: Fails on prereleases; https://github.com/mindsers/changelog-reader-action/pull/39
       - name: Parse changelog
         id: changelog
         uses: mindsers/changelog-reader-action@v2
@@ -97,7 +98,7 @@ jobs:
             ## ${{ steps.changelog.outputs.version }} - ${{ steps.changelog.outputs.date }}
 
             ${{ steps.changelog.outputs.changes }}
-          draft: true
+          draft: false
           prerelease: ${{ steps.changelog.outputs.status == 'prereleased' }}
 
       - name: Create Sentry release

.gitignore

@@ -29,4 +29,6 @@
 
 # Docs
 !docs/**
-docs/_build
+docs/_build
+
+dipdup_indexer

CHANGELOG.md

@@ -4,7 +4,7 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog], and this project adheres to [Semantic Versioning].
 
-## [Unreleased]
+## [7.0.0rc4] - 2023-08-23
 
 ### Added
 
@@ -17,9 +17,9 @@ The format is based on [Keep a Changelog], and this project adheres to [Semantic
 - cli: Fixed `schema wipe` command crash due to `dipdup_meta` table being always immune.
 - config: Don't create empty SentryConfig if DSN is not set.
 - context: Share internal state between context instances.
+- evm.node: Fixed keepalive loop for websocket connection.
+- evm.node: Fixed parsing empty realtime message payloads.
 - jobs: Don't add jobs before scheduler is started.
-- node: Fixed keepalive loop for websocket connection.
-- node: Fixed parsing empty realtime message payloads.
 - package: Fixed package detection for poetry managed projects.
 - package: Fixed mypy command in default template.
 - package: Create package symlink only when needed.
@@ -1141,7 +1141,8 @@ This release contains no changes except for the version number.
 [semantic versioning]: https://semver.org/spec/v2.0.0.html
 
 <!-- Versions -->
-[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc3...HEAD
+[Unreleased]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc4...HEAD
+[7.0.0rc4]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc3...7.0.0rc4
 [7.0.0rc3]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc2...7.0.0rc3
 [6.5.10]: https://github.com/dipdup-io/dipdup/compare/6.5.9...6.5.10
 [7.0.0rc2]: https://github.com/dipdup-io/dipdup/compare/7.0.0rc1...7.0.0rc2

docs/0.quickstart-evm.md

@@ -0,0 +1,187 @@
+---
+title: "Quickstart on EVM"
+description: "This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details."
+navigation.icon: "stars"
+network: "ethereum"
+---
+
+# Quickstart
+
+This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details.
+
+Let's create an indexer for the [USDt token contract](https://etherscan.io/address/0xdac17f958d2ee523a2206206994597c13d831ec7). Our goal is to save all token transfers to the database and then calculate some statistics of its holders' activity.
+
+## Install DipDup
+
+A modern Linux/macOS distribution with Python 3.11 installed is required to run DipDup.
+
+We have a [convenient installer](https://dipdup.io/install.py) that will install DipDup and all the required dependencies for you. It uses pipx to install DipDup as a CLI application and make it available everywhere for the current user. Run the following command in your terminal:
+
+```shell [Terminal]
+curl -Lsf https://dipdup.io/install.py | python3
+```
+
+If you don't want to use our script, install DipDup manually using commands from the snippet below. You can use any Python package manager you like, but we recommend [PDM](https://pdm.fming.dev/latest/) and provide some useful scripts for it. Here are some spells to get you started:
+
+```shell [Terminal]
+# pipx
+pipx install dipdup datamodel-code-generator
+
+# PDM
+pdm init --python 3.11 --lib  # use "">=3.11,<3.12" for requires-python
+pdm venv create
+pdm add "dipdup>=7.0.0rc4,<8" --venv
+$(pdm venv activate)
+
+# Poetry
+poetry init --python ">=3.11,<3.12"
+poetry add "dipdup>=7.0.0rc4,<8"
+poetry shell
+
+# pip + venv
+python -m venv .venv
+. .venv/bin/activate
+echo "dipdup>=7.0.0rc4,<8" > requirements.txt
+pip install -r requirements.txt -e .
+```
+
+## Create a project
+
+DipDup CLI has a built-in project generator. Run the following command in your terminal:
+
+```shell [Terminal]
+dipdup new
+```
+
+For educational purposes, we'll create a project from scratch, so choose `[none]` network and `demo_blank` template.
+
+::banner{type="note"}
+Want to skip a tutorial? Choose `EVM-compatible` and `demo_evm_events` instead!
+::
+
+Follow the instructions; the project will be created in the new directory.
+
+## Write a configuration file
+
+In the project root, you'll find a file named `dipdup.yaml`. It's the main configuration file of your indexer. We will discuss it in detail in the [Config](1.getting_started/3.config.md) section; for now just replace its content with the following:
+
+```yaml [dipdup.yaml]
+{{ #include ../src/demo_evm_events/dipdup.yaml }}
+```
+
+## Generate types and stubs
+
+Now it's time to generate typeclasses and callback stubs based on definitions from config. Examples below use `demo_evm_events` as a package name; yours may differ.
+
+Run the following command:
+
+```shell [Terminal]
+dipdup init
+```
+
+DipDup will create a Python package `demo_evm_events` with everything you need to start writing your indexer. Use `package tree` command to see the generated structure:
+
+```shell [Terminal]
+$ dipdup package tree
+demo_evm_events [.]
+├── abi
+│   ├── eth_usdt/abi.json
+│   └── eth_usdt/events.json
+├── configs
+│   ├── dipdup.compose.yaml
+│   ├── dipdup.sqlite.yaml
+│   ├── dipdup.swarm.yaml
+│   └── replay.yaml
+├── deploy
+│   ├── .env.default
+│   ├── Dockerfile
+│   ├── compose.sqlite.yaml
+│   ├── compose.swarm.yaml
+│   ├── compose.yaml
+│   ├── sqlite.env.default
+│   └── swarm.env.default
+├── graphql
+├── handlers
+│   └── on_transfer.py
+├── hasura
+├── hooks
+│   ├── on_index_rollback.py
+│   ├── on_reindex.py
+│   ├── on_restart.py
+│   └── on_synchronized.py
+├── models
+│   └── __init__.py
+├── sql
+├── types
+│   └── eth_usdt/evm_events/transfer.py
+└── py.typed
+```
+
+That's a lot of files and directories! But don't worry, we will need only `models` and `handlers` sections in this guide.
+
+## Define data models
+
+DipDup supports storing data in SQLite, PostgreSQL and TimescaleDB databases. We use custom ORM based on Tortoise ORM as an abstraction layer.
+
+First, you need to define a model class. Our schema will consist of a single model `Holder` with the following fields:
+
+|             |                                     |
+| ----------- | ----------------------------------- |
+| `address`   | account address                     |
+| `balance`   | token amount held by the account    |
+| `turnover`  | total amount of transfer/mint calls |
+| `tx_count`  | number of transfers/mints           |
+| `last_seen` | time of the last transfer/mint      |
+
+Here's how to implement this model in DipDup:
+
+```python [models/__init__.py]
+{{ #include ../src/demo_evm_events/models/__init__.py }}
+```
+
+## Implement handlers
+
+Everything's ready to implement an actual indexer logic.
+
+Our task is to index all the balance updates. Put some code to the `on_transfer` handler callback to process matched logs:
+
+```python [handlers/on_transfer.py]
+{{ #include ../src/demo_evm_events/handlers/on_transfer.py }}
+```
+
+And that's all! We can run the indexer now.
+
+## Next steps
+
+Run the indexer in-memory:
+
+```bash
+dipdup run
+```
+
+Store data in SQLite database:
+
+```bash
+dipdup -c . -c configs/dipdup.sqlite.yaml run
+```
+
+Or spawn a docker-compose stack:
+
+```bash
+cd deploy
+cp .env.default .env
+# Edit .env before running
+docker-compose up
+```
+
+DipDup will fetch all the historical data and then switch to realtime updates. You can check the progress in the logs.
+
+If you use SQLite, run a query to check the data:
+
+```bash
+sqlite3 demo_evm_events.sqlite 'SELECT * FROM holder LIMIT 10'
+```
+
+If you run a Compose stack, check open `http://127.0.0.1:8080` in your browser to see the Hasura console (exposed port may differ). You can use it to explore the database and build GraphQL queries.
+
+Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn more about DipDup configuration and features.

docs/0.quickstart.md → docs/0.quickstart-tezos.md

@@ -1,7 +1,8 @@
 ---
-title: "Quickstart"
+title: "Quickstart on Tezos"
 description: "This page will guide you through the steps to get your first selective indexer up and running in a few minutes without getting too deep into the details."
 navigation.icon: "stars"
+network: "tezos"
 ---
 
 # Quickstart
@@ -29,18 +30,18 @@ pipx install dipdup datamodel-code-generator
 # PDM
 pdm init --python 3.11 --lib  # use "">=3.11,<3.12" for requires-python
 pdm venv create
-pdm add "dipdup>=7.0.0rc3,<8" --venv
+pdm add "dipdup>=7.0.0rc4,<8" --venv
 $(pdm venv activate)
 
 # Poetry
 poetry init --python ">=3.11,<3.12"
-poetry add "dipdup>=7.0.0rc3,<8"
+poetry add "dipdup>=7.0.0rc4,<8"
 poetry shell
 
 # pip + venv
 python -m venv .venv
 . .venv/bin/activate
-echo "dipdup>=7.0.0rc3,<8" > requirements.txt
+echo "dipdup>=7.0.0rc4,<8" > requirements.txt
 pip install -r requirements.txt -e .
 ```
 
@@ -195,4 +196,4 @@ sqlite3 demo_token.sqlite 'SELECT * FROM holder LIMIT 10'
 
 If you run a Compose stack, check open `http://127.0.0.1:8080` in your browser to see the Hasura console (exposed port may differ). You can use it to explore the database and build GraphQL queries.
 
-Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn in-depth about DipDup configuration and features.
+Congratulations! You've just created your first DipDup indexer. Proceed to the Getting Started section to learn more about DipDup configuration and features.

docs/1.getting-started/1.installation.md

@@ -48,7 +48,7 @@ PDM is a very powerful package manager with a lot of features. Also, it can run
 ```shell [Terminal]
 pdm init --python 3.11 --lib  # use "">=3.11,<3.12" for requires-python
 pdm venv create
-pdm add "dipdup>=7.0.0rc3,<8" --venv
+pdm add "dipdup>=7.0.0rc4,<8" --venv
 $(pdm venv activate)
 ```
 
@@ -58,7 +58,7 @@ Poetry is another popular tool to manage Python projects. It doesn't support scr
 
 ```shell [Terminal]
 poetry init --python ">=3.11,<3.12"
-poetry add "dipdup>=7.0.0rc3,<8"
+poetry add "dipdup>=7.0.0rc4,<8"
 poetry shell
 ```
 
@@ -69,7 +69,7 @@ Finally, if you prefer to do everything manually, you can use pip. It's the most
 ```shell [Terminal]
 python -m venv .venv
 . .venv/bin/activate
-echo "dipdup>=7.0.0rc3,<8" >> requirements.txt
+echo "dipdup>=7.0.0rc4,<8" >> requirements.txt
 pip install -r requirements.txt -e .
 ```
 

docs/2.indexes/2.tezos_tzkt_big_maps.md

@@ -13,7 +13,7 @@ Big maps are lazy structures allowing to access and update only exact keys. Gas
 ```yaml [dipdup.yaml]
 indexes:
   token_big_map_index:
-    kind: tezos.tzkt.big_map
+    kind: tezos.tzkt.big_maps
     datasource: tzkt
     skip_history: never
     handlers: