Typescript indexer-grpc client

.github/workflows/lint.yaml

@@ -0,0 +1,30 @@
+name: "Lint"
+on:
+  # Allow us to run this specific workflow without a PR
+  workflow_dispatch:
+  pull_request:
+  push:
+    branches:
+      - main
+
+# cancel redundant builds
+concurrency:
+  # for push and workflow_dispatch events we use `github.sha` in the concurrency group and don't really cancel each other out/limit concurrency
+  # for pull_request events newer jobs cancel earlier jobs to save on CI etc.
+  group: ${{ github.workflow }}-${{ github.event_name }}-${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && github.sha || github.head_ref || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  Python:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: python
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4 
+      - uses: snok/install-poetry@v1
+        with:
+          version: 1.4.2
+      - run: poetry install
+      - run: poetry run poe lint

.gitignore

@@ -10,3 +10,6 @@ __pycache__.DS_Store
 __pycache__
 config.yaml
 cursor.txt
+node_modules
+build
+venv

.python-version

@@ -0,0 +1 @@
+3.11

README.md

@@ -22,45 +22,6 @@ mainnet: 34.30.218.153:50051
 - The response is a stream of `RawDatastreamResponse` objects.
 - For each supported language, there is an `aptos` folder which contains the auto-generate protobuf files in that language. You can check out the files to see the stream response format and figure out how to parse the response. 
 
-# Quickstart
-## Python
-### Prerequisite
-- Python 3.7 or higher
-- `pip` version 9.0.1 or higher
-### Guide
-1. Install the latest version of gRPC and tooling for Python:
-  ```
-  python -m pip install grpcio
-  python -m pip install grpcio-tools
-
-  ```
-2. Download the example:
-```
-# Clone the repository to get the example code:
-$ git clone https://github.com/aptos-labs/aptos-indexer-client-examples
-# Navigate to the python folder
-$ cd aptos-indexer-client-examples/python
-```
-In this example, we are creating an event parser.
-3. Create a client.
-   - First you need to create a gRPC client that reads the stream of data.
-   - We've create an example client in `grpc_client.py`. This client
-     - Connects to the gRPC server and reads a stream of transaction data.
-     - Calls the function `parse` to parse the transaction
-     - Validates the chain ID and transaction version.
-4. Create a parser.
-   - In `grpc_parser.py`, we have implemented a `parse` function which accepts a `Transaction` as a parameter.
-   - The example code shows how to implement custom filtering and parse a `Transaction` and the associated `Event`'s.
-   - The function returns t.
-5. Insert data rows into database.
-   - In the example, we use Postgres for the database and SQLAlchemy as the ORM. To run the example code, install the following:
-     ```
-     python -m pip install psycopg2
-     python -m pip install sqlalchemy
-     ```
-   - In `grpc_client.py`, after the events are parsed, all the event objects are then added to the database.
-6. Run `python grpc_client.py` to start indexing! 
-
 ## Typescript / Node
 ### Prerequisite
 - `node`: This requires Node 0.12.x or greater.

python/Dockerfile

@@ -15,4 +15,4 @@ RUN poetry config virtualenvs.create false \
 COPY *.py /app/
 COPY /aptos /app/aptos
 
-CMD ["poetry", "run", "python", "grpc_client.py", "--config", "/app/config/config.yaml"]
+CMD ["poetry", "run", "python", "processor.py", "--config", "/app/config/config.yaml"]

python/README.md

@@ -0,0 +1,58 @@
+## Python Quickstart
+### Prerequisite
+- Python 3.7 or higher
+- `pip` version 9.0.1 or higher
+### Tutorial
+1. Install the latest version of gRPC and tooling for Python:
+  ```
+  python -m pip install grpcio
+  python -m pip install grpcio-tools
+
+  ```
+2. Download the example:
+```
+# Clone the repository to get the example code:
+$ git clone https://github.com/aptos-labs/aptos-indexer-client-examples
+# Navigate to the python folder
+$ cd aptos-indexer-client-examples/python
+```
+In this example, we are creating an event parser.
+3. Create a processer.
+   - First you need to create an indexer processor that reads the stream of data.
+   - We've create an example client in `processor.py`. This client
+     - Connects to the gRPC server and reads a stream of transaction data.
+     - Calls the function `parse` to parse the transaction
+     - Validates the chain ID and transaction version.
+4. Create a parser.
+   - In `grpc_parser.py`, we have implemented a `parse` function which accepts a `Transaction` as a parameter.
+   - The example code shows how to implement custom filtering and parse a `Transaction` and the associated `Event`'s.
+   - The function returns t.
+5. Insert data rows into database.
+   - In the example, we use Postgres for the database and SQLAlchemy as the ORM. To run the example code, install the following:
+     ```
+     python -m pip install psycopg2
+     python -m pip install sqlalchemy
+     ```
+   - In `grpc_client.py`, after the events are parsed, all the event objects are then added to the database.
+6. Run `python grpc_client.py` to start indexing! 
+
+## Development
+
+### Install all dependencies
+
+```bash
+poetry install
+```
+
+### Linting & autoformatting
+
+```bash
+poetry run poe pyright # typecheck
+poetry run poe format # autoformat via black
+```
+
+### Run locally in Docker
+
+```bash
+docker compose up --build --force-recreate
+```

python/aptos/indexer/v1/raw_data_pb2.pyi

@@ -0,0 +1,23 @@
+from aptos.transaction.testing1.v1 import transaction_pb2 as _transaction_pb2
+from google.protobuf.internal import containers as _containers
+from google.protobuf import descriptor as _descriptor
+from google.protobuf import message as _message
+from typing import ClassVar as _ClassVar, Iterable as _Iterable, Mapping as _Mapping, Optional as _Optional, Union as _Union
+
+DESCRIPTOR: _descriptor.FileDescriptor
+
+class GetTransactionsRequest(_message.Message):
+    __slots__ = ["starting_version", "transactions_count"]
+    STARTING_VERSION_FIELD_NUMBER: _ClassVar[int]
+    TRANSACTIONS_COUNT_FIELD_NUMBER: _ClassVar[int]
+    starting_version: int
+    transactions_count: int
+    def __init__(self, starting_version: _Optional[int] = ..., transactions_count: _Optional[int] = ...) -> None: ...
+
+class TransactionsResponse(_message.Message):
+    __slots__ = ["chain_id", "transactions"]
+    CHAIN_ID_FIELD_NUMBER: _ClassVar[int]
+    TRANSACTIONS_FIELD_NUMBER: _ClassVar[int]
+    chain_id: int
+    transactions: _containers.RepeatedCompositeFieldContainer[_transaction_pb2.Transaction]
+    def __init__(self, transactions: _Optional[_Iterable[_Union[_transaction_pb2.Transaction, _Mapping]]] = ..., chain_id: _Optional[int] = ...) -> None: ...