scripts start observers #178

miiu96 · 2022-10-06T15:43:43Z

Added scripts that will start 4 observers and 4 indexer binaries that can index data from testnet, devnet or mainnet based on the .env config file
Refactor and improve how the received data is processed (altered accounts are no longer extracted on the indexer side)

andreibancioiu · 2022-10-07T08:43:26Z

scripts/observers/clean.py

+
+try:
+    shutil.rmtree(working_dir)
+    print("done")


print(f"removed directory: {working_dir}")

andreibancioiu · 2022-10-07T08:44:33Z

scripts/observers/clean.py

+    shutil.rmtree(working_dir)
+    print("done")
+except:
+    print('did nothing')


"nothing to clean" / "directory not removed" or simply pass.

andreibancioiu · 2022-10-07T08:46:32Z

scripts/observers/config.py

+    update_toml_node(node_config, shard_id)
+    update_toml_indexer(indexer_config, shard_id)
+
+


Usually, we define a main(), then call it at the end of the file.

Examples:

https://github.com/ElrondNetwork/db-archive-scripts/tree/main/archive_py

https://github.com/ElrondNetwork/elrond-sdk-images/tree/main/scripts

https://github.com/ElrondNetwork/rosetta-docker-scripts

(optional, can also be left as it is)

added main for all files

andreibancioiu · 2022-10-07T08:47:39Z

scripts/observers/config.py

+    data1 = toml.load(path+"/prefs.toml")
+    data1['config']['web-socket']['server-url'] = str(shard_id)
+    if shard_id != METACHAIN:
+        data1['config']['web-socket']['server-url'] = "localhost:" + str(22111+shard_id)


Is 22111 the port base? Perhaps define a constant or receive it in the env file?

added a constant

andreibancioiu · 2022-10-07T08:48:45Z

scripts/observers/config.py

+
+def update_toml_indexer(path, shard_id):
+    # prefs.toml
+    data1 = toml.load(path+"/prefs.toml")


Bit of renaming: data1 vs. prefs_data or something similar.

andreibancioiu · 2022-10-07T08:49:55Z

scripts/observers/config.py

+
+def update_toml_indexer(path, shard_id):
+    # prefs.toml
+    data1 = toml.load(path+"/prefs.toml")


If path is a pathlib.Path, then you can also use the / operator. For example: https://github.com/ElrondNetwork/elrond-sdk-images/blob/main/scripts/build_contract_rust_within_docker.py#L104

refactored.

andreibancioiu · 2022-10-07T08:54:51Z

scripts/observers/config.py

+import sys
+import shutil
+from pathlib import Path
+from git import Repo


This does not seem from the standard library. Perhaps add a requirements.txt file and a README explanation?

For example:

https://github.com/ElrondNetwork/db-archive-scripts/blob/main/requirements.txt

https://github.com/ElrondNetwork/db-archive-scripts/blob/main/README.md#prerequisites

andreibancioiu · 2022-10-07T08:56:54Z

scripts/observers/config.py

+# build binary indexer
+print("building indexer...")
+os.chdir("../../cmd/elasticindexer")
+os.system("go build")


Usually, we favor subprocess instead of os.system. Example:

https://github.com/ElrondNetwork/rosetta-ci/blob/v1.0.0/scripts/code_compile.py#L22

(rosetta-ci is a private repository)

andreibancioiu · 2022-10-07T08:57:30Z

scripts/observers/config.py

+
+# prepare observers
+print("preparing config...")
+prepare_observer(0)


andreibancioiu · 2022-10-07T08:58:19Z

scripts/observers/start.py

+    os.chdir(working_dir_observer + "/node")
+    command = "./node" + " --log-level *:DEBUG --no-key --log-save"
+    # os.system("gnome-terminal 'bash -c \"" + command + ";bash\"'")
+    os.system("screen -d -m -S obs" + str(shard_id) + " " + command)


Oh, I see, we're using screen. Perhaps get a second opinion from @valentin-lup, if this is all right.

andreibancioiu · 2022-10-07T08:59:17Z

scripts/observers/stop.py

@@ -0,0 +1,4 @@
+import os
+
+os.system("killall screen")


Will kill all screen instances that the user has. Perhaps we should find another solution for this, or remove the stop.py script in order to avoid undesired effects?

bogdan-rosianu · 2022-10-27T07:34:47Z

process/elasticproc/logsevents/interface.go

-	updatePropNFT   *data.NFTDataUpdate
+	tokenInfo     *data.TokenInfo
+	delegator     *data.Delegator
+	processed     bool


move processed bool to the last position

bogdan-rosianu · 2022-10-27T10:10:50Z

scripts/observers/.env

+ELROND_GO_BRANCH="feat/altered-accounts"
+
+WORKING_DIRECTORY="IndexerObservers"
+OBSERVER_DIR="observer_shard_"


might rename to OBSERVER_DIR_PREFIX

bogdan-rosianu · 2022-10-27T10:29:18Z

scripts/observers/clean.py

+    working_dir = Path(Path.home() / os.getenv('WORKING_DIRECTORY'))
+
+    try:
+        shutil.rmtree(working_dir)


use a default path for the scripts, not the user's home

bogdan-rosianu · 2022-10-27T12:02:31Z

scripts/observers/config.py

@@ -80,7 +80,11 @@ def prepare_observer(shard_id, working_dir, config_folder):

 def main():
    load_dotenv()
-    working_dir = Path(Path.home() / os.getenv('WORKING_DIRECTORY'))
+    working_dir_var = os.getenv('WORKING_DIRECTORY')
+    if working_dir_var == "":


duplicated code. might extract a function an use it in the both places

andreibancioiu · 2022-10-27T13:31:02Z

scripts/observers/start.py


    current_directory = os.getcwd()
    # start observer
-    os.chdir(working_dir_observer + "/node")
+    os.chdir(Path(working_dir_observer / "node"))


The cast to Path is not required (here and below).

andreibancioiu · 2022-10-27T13:31:40Z

scripts/observers/start.py

 from dotenv import load_dotenv
+from utils import *


Usually, we should be importing modules without using the wildcard *.

andreibancioiu · 2022-10-27T13:32:06Z

scripts/observers/config.py

+
+    # build binary elrond-go
+    print("building node...")
+    subprocess.check_call(["go", "build"], cwd=Path(elrond_go_folder / "cmd/node"))


miiu96 added 3 commits October 6, 2022 18:42

push

91059d1

extra checks and prints

a1edbfa

small fix

63e3991

andreibancioiu reviewed Oct 7, 2022

View reviewed changes

miiu96 added 7 commits October 25, 2022 10:56

remove altered accounts part

88fea1f

fix integration tests

2108634

refactor python scrips

88ffeeb

small fix

8346462

fix unmarshall

b3997bd

fix log processor

6574353

refactor

5b1c19f

miiu96 self-assigned this Oct 27, 2022

miiu96 marked this pull request as ready for review October 27, 2022 07:10

bogdan-rosianu reviewed Oct 27, 2022

View reviewed changes

fixes after review

2cd634f

bogdan-rosianu reviewed Oct 27, 2022

View reviewed changes

fixes

c1d9520

bogdan-rosianu previously approved these changes Oct 27, 2022

View reviewed changes

refactor

833d50a

miiu96 dismissed bogdan-rosianu’s stale review via 833d50a October 27, 2022 12:44

bogdan-rosianu previously approved these changes Oct 27, 2022

View reviewed changes

andreibancioiu previously approved these changes Oct 27, 2022

View reviewed changes

update elrond go core

3fed07b

miiu96 dismissed stale reviews from andreibancioiu and bogdan-rosianu via 3fed07b October 27, 2022 13:42

miiu96 added 2 commits October 27, 2022 16:57

remove Path cast

b57115b

fix integration tests

0359f0c

bogdan-rosianu approved these changes Oct 27, 2022

View reviewed changes

andreibancioiu approved these changes Oct 27, 2022

View reviewed changes

miiu96 merged commit dc2a42c into feat/altered-accounts Oct 27, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

scripts start observers #178

scripts start observers #178

miiu96 commented Oct 6, 2022 •

edited

Loading

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

andreibancioiu Oct 7, 2022

andreibancioiu Oct 7, 2022

andreibancioiu Oct 7, 2022

miiu96 Oct 26, 2022

bogdan-rosianu Oct 27, 2022

miiu96 Oct 27, 2022

bogdan-rosianu Oct 27, 2022

miiu96 Oct 27, 2022

bogdan-rosianu Oct 27, 2022

miiu96 Oct 27, 2022

bogdan-rosianu Oct 27, 2022

miiu96 Oct 27, 2022

andreibancioiu Oct 27, 2022

miiu96 Oct 27, 2022

andreibancioiu Oct 27, 2022

andreibancioiu Oct 27, 2022

		update_toml_node(node_config, shard_id)
		update_toml_indexer(indexer_config, shard_id)

scripts start observers #178

scripts start observers #178

Conversation

miiu96 commented Oct 6, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

miiu96 commented Oct 6, 2022 •

edited

Loading